Implement non-greedy tokenizer that tries to maximize token lengths by thement · Pull Request #242 · ggml-org/llama.cpp

thement · 2023-03-17T16:36:53Z

No description provided.

Piezoid · 2023-03-17T16:57:38Z

Although I haven't examined the code, I've tested it on several prompts and can already conclude that this patch allows Llama to write in French.
Previously it spouted nonsense before falling back to english.

Piezoid · 2023-03-17T17:30:04Z

Is the space meant to be a separate token? I noticed that it often get fused with the first user provided token.

It should be fused to the first token! This is how original python llama code parses it.
I can dig out more details if you want.

ggerganov

Merge it if results look ok.
I won't be able to have detailed look in the next few days

- this is to match original llama tokenizer behavior

…gml-org#242) * Implement non-greedy tokenizer that tries to maximize token lengths * Insert single space in front of the prompt - this is to match original llama tokenizer behavior --------- Co-authored-by: Jakub Horak <jakub.horak@ibawizard.net>

…16 resid loss) ggml-org#242: when F3BEST_QDUMP is active, host zero-fills the resid region in the XR bundle. nm tile's add_bf16 then computes F = O(Q) + 0 = Q, so the s-region holds an exact bf16 Q (no bf16(Q+R)-minus-R precision loss that made QDUMP inaccurate per BRAINSTORM ggml-org#3). Result (3B probe): |Q+resid|==|Q| (resid=0 confirmed), rel_err=10.3 vs CPU Q = the expected sqrt(128) prescale mismatch -> Q-GEMV+RoPE CORRECT for 3B. This is the clean bisection of the Q-path: H1 (Q-path wrong) DISPROVEN without resid corruption. H2 (within-group q-head permutation in attn_out/O-proj gather) remains the primary hypothesis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

DECISIVE Q-path bisection (R6 §9.1). The existing [qdump-cpuq] golden compares NPU-Q against a hand-rolled cpuq (manual Wq-dequant + manual interleaved RoPE), which the [f3best-ref] staging reference proved broken (relOp_fromNormed=0.95 vs relOp_ref=0.0045 from native rnode). The hand-rolled cpuq also applies the 1/sqrt(head_dim) prescale, so the tap (which reads raw pre-prescale Q, per ggml-org#242) shows a spurious sqrt(128) ratio — rel_err=10.3. [qdump-rnode] compares NPU-Q (s-region, raw, no prescale) directly against ggml's native roped-Q tensor rope_node->data (also raw, no prescale), element-wise via h*nb[1], for the first attn-group (ag_f heads). 3B probe result (hd=128, layer 0): [qdump-rnode] seq=0 ag=3 hd=128 cmp=384 nan=0 rel_err=0.008145 npu_max=14.0601 rnode_max=14.0906 mean_ratio=1.01248 rel_err=0.008 / mean_ratio=1.012 = bf16-precision + benign 1.2% prescale residue. NPU-Q matches the native roped-Q. Q-GEMV + RoPE are CORRECT against the only trustworthy golden. The bug is INSIDE flowkv (score/softmax/value), not above it (not Q-GEMV / RoPE / LUT / Q delivery). Probe-mode only: the CPU graph runs fully before dispatch, so rope_node->data is filled; live mode skips CPU ops so rnode->data is stale (qdump-rnode guards rope_node->type == F32 and only runs under F3BEST_QDUMP). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve.

…g#242)

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve. Fold (r208): RDNA2 (gfx1030) FA prefill, reported and validated by @slavap (PR ggml-org#270/ggml-org#271; Radeon Pro V620 gfx1030, Qwen3.6-35B-A3B IQ4_XS, -b 8192 -ub 2048 -fa 1). This is the RDNA2 long-context deficit of issue ggml-org#214, and it is folded here rather than cut as new blocks. - q_global threshold 16 -> 64: the two kernel-side conditions, the Q_tmp_count stub and all seven launcher q_h2 arguments move together (a launcher that omits the argument is the r88 null-Q_h2 fault). At ncols 16..32 the global-half2 Q path re-reads Q per K-tile pass and the checked tail spilled ~180 VGPRs; with Q back in LDS it does not. - rdna_cap_ncols2 lifted for DKQ == 256 (D=128 keeps the cap). The cap dates from r76, when HIP's RDNA2 occupancy query returned 0 for the wide D=256 variants; launch_fattn computes RDNA2 occupancy itself now, so the cap is stale. Checked against the tree's own formula (4*waves_per_simd/waves_per_block, LDS cap 128K/lds) over the patched gfx1030 HSACO: every D=256 shape the launcher can now pick reports >= 2 blocks, so the GGML_ASSERT(max_blocks_per_sm > 0) abort the cap guarded against is unreachable. - the checked tail clamps out-of-range rows to the last in-bounds row instead of branching to a local zero array. The tail values are discarded by the k_VKQ_sup guards, every read stays inside the cache, and the pointer-select the branch existed to avoid (the kv=113 flat-scratch fault) is gone with it. i_sup - 1 may be -1 (a V chunk wholly past the KV end): row -1 is the last VALID row, and every weight in that chunk is zero, so the comment in the loader says so and warns against "fixing" it to max(i_sup - 1, 0), which would read one row past the cache. Verified here on ROCm 10.1.0 against this tree: both levers compile for gfx1030 (all 12 template-instances tile TUs, no warnings), the D=256 4x8/16x2 checked-tail scratch goes 480/500 B -> 0 B (the submitter's 488/520 on their toolchain), the largest static LDS in the whole tile family stays 63488 B of gfx1030's 65536 B, and for gfx1201 every tile kernel's machine code is bit-identical before and after (88/88 and 106/106 kernels hashed per kernel), so RDNA4, RDNA3_0, RDNA3_5, NVIDIA and MUSA/SYCL keep their codegen and numerics exactly. The timings stay the submitter's (pp2048 @ d32768 1041.0 -> 1259.8 t/s, tg256 @ d131072 50.9 -> 62.4 across their two arms); RDNA2 is best-effort per CONTAINERS.md and carries no gate in this repo. Not taken from the PRs: their own patches-rdna2-r205/ + release-rdna2-r205.json + scripts/apply-rdna2-r205.sh re-packaging (the same call as r88), and the block 16 two-lane prefill the branches were stacked on (PR ggml-org#268, a separate candidate).

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve. Fold (r208): RDNA2 (gfx1030) FA prefill, reported and validated by @slavap (PR ggml-org#270/ggml-org#271; Radeon Pro V620 gfx1030, Qwen3.6-35B-A3B IQ4_XS, -b 8192 -ub 2048 -fa 1). This is the RDNA2 long-context deficit of issue ggml-org#214, and it is folded here rather than cut as new blocks. - q_global threshold 16 -> 64: the two kernel-side conditions, the Q_tmp_count stub and all seven launcher q_h2 arguments move together (a launcher that omits the argument is the r88 null-Q_h2 fault). At ncols 16..32 the global-half2 Q path re-reads Q per K-tile pass and the checked tail spilled ~180 VGPRs; with Q back in LDS it does not. - rdna_cap_ncols2 lifted for DKQ == 256 (D=128 keeps the cap). The cap dates from r76, when HIP's RDNA2 occupancy query returned 0 for the wide D=256 variants; launch_fattn computes RDNA2 occupancy itself now, so the cap is stale. Checked against the tree's own formula (4*waves_per_simd/waves_per_block, LDS cap 128K/lds) over the patched gfx1030 HSACO: every D=256 shape the launcher can now pick reports >= 2 blocks, so the GGML_ASSERT(max_blocks_per_sm > 0) abort the cap guarded against is unreachable. - the checked tail clamps out-of-range rows to the last in-bounds row instead of branching to a local zero array. The tail values are discarded by the k_VKQ_sup guards, every read stays inside the cache, and the pointer-select the branch existed to avoid (the kv=113 flat-scratch fault) is gone with it. i_sup - 1 may be -1 (a V chunk wholly past the KV end): row -1 is the last VALID row, and every weight in that chunk is zero, so the comment in the loader says so and warns against "fixing" it to max(i_sup - 1, 0), which would read one row past the cache. Verified here on ROCm 10.1.0 against this tree: both levers compile for gfx1030 (all 12 template-instances tile TUs, no warnings), the D=256 4x8/16x2 checked-tail scratch goes 480/500 B -> 0 B (the submitter's 488/520 on their toolchain), the largest static LDS in the whole tile family stays 63488 B of gfx1030's 65536 B, and for gfx1201 every tile kernel's machine code is bit-identical before and after (88/88 and 106/106 kernels hashed per kernel), so RDNA4, RDNA3_0, RDNA3_5, NVIDIA and MUSA/SYCL keep their codegen and numerics exactly. The timings stay the submitter's (pp2048 @ d32768 1041.0 -> 1259.8 t/s, tg256 @ d131072 50.9 -> 62.4 across their two arms); RDNA2 is best-effort per CONTAINERS.md and carries no gate in this repo. Not taken from the PRs: their own patches-rdna2-r205/ + release-rdna2-r205.json + scripts/apply-rdna2-r205.sh re-packaging (the same call as r88), and the block 16 two-lane prefill the branches were stacked on (PR ggml-org#268, a separate candidate).

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve. Fold (r208): RDNA2 (gfx1030) FA prefill, reported and validated by @slavap (PR ggml-org#270/ggml-org#271; Radeon Pro V620 gfx1030, Qwen3.6-35B-A3B IQ4_XS, -b 8192 -ub 2048 -fa 1). This is the RDNA2 long-context deficit of issue ggml-org#214, and it is folded here rather than cut as new blocks. - q_global threshold 16 -> 64: the two kernel-side conditions, the Q_tmp_count stub and all seven launcher q_h2 arguments move together (a launcher that omits the argument is the r88 null-Q_h2 fault). At ncols 16..32 the global-half2 Q path re-reads Q per K-tile pass and the checked tail spilled ~180 VGPRs; with Q back in LDS it does not. - rdna_cap_ncols2 lifted for DKQ == 256 (D=128 keeps the cap). The cap dates from r76, when HIP's RDNA2 occupancy query returned 0 for the wide D=256 variants; launch_fattn computes RDNA2 occupancy itself now, so the cap is stale. Checked against the tree's own formula (4*waves_per_simd/waves_per_block, LDS cap 128K/lds) over the patched gfx1030 HSACO: every D=256 shape the launcher can now pick reports >= 2 blocks, so the GGML_ASSERT(max_blocks_per_sm > 0) abort the cap guarded against is unreachable. - the checked tail clamps out-of-range rows to the last in-bounds row instead of branching to a local zero array. The tail values are discarded by the k_VKQ_sup guards, every read stays inside the cache, and the pointer-select the branch existed to avoid (the kv=113 flat-scratch fault) is gone with it. i_sup - 1 may be -1 (a V chunk wholly past the KV end): row -1 is the last VALID row, and every weight in that chunk is zero, so the comment in the loader says so and warns against "fixing" it to max(i_sup - 1, 0), which would read one row past the cache. Verified here on ROCm 10.1.0 against this tree: both levers compile for gfx1030 (all 12 template-instances tile TUs, no warnings), the D=256 4x8/16x2 checked-tail scratch goes 480/500 B -> 0 B (the submitter's 488/520 on their toolchain), the largest static LDS in the whole tile family stays 63488 B of gfx1030's 65536 B, and for gfx1201 every tile kernel's machine code is bit-identical before and after (88/88 and 106/106 kernels hashed per kernel), so RDNA4, RDNA3_0, RDNA3_5, NVIDIA and MUSA/SYCL keep their codegen and numerics exactly. The timings stay the submitter's (pp2048 @ d32768 1041.0 -> 1259.8 t/s, tg256 @ d131072 50.9 -> 62.4 across their two arms); RDNA2 is best-effort per CONTAINERS.md and carries no gate in this repo. Not taken from the PRs: their own patches-rdna2-r205/ + release-rdna2-r205.json + scripts/apply-rdna2-r205.sh re-packaging (the same call as r88), and the block 16 two-lane prefill the branches were stacked on (PR ggml-org#268, a separate candidate).

thement mentioned this pull request Mar 17, 2023

Differences with the llama tokenizer #167

Closed

thement marked this pull request as ready for review March 17, 2023 17:14

j-f1 reviewed Mar 17, 2023

View reviewed changes

Comment thread utils.cpp Resolved

Piezoid reviewed Mar 17, 2023

View reviewed changes

ggerganov approved these changes Mar 17, 2023

View reviewed changes

jxhor added 2 commits March 17, 2023 18:43

Implement non-greedy tokenizer that tries to maximize token lengths

7e1041a

Insert single space in front of the prompt

7566d1a

- this is to match original llama tokenizer behavior

thement merged commit c9f670a into ggml-org:master Mar 17, 2023

ivanstepanovftw mentioned this pull request May 4, 2023

Avoid hardcoding a space at the beginning of the prompt. #1315

Closed

Bearsaerker mentioned this pull request Mar 12, 2025

Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352

Closed

pwilkin pushed a commit to pwilkin/llama.cpp that referenced this pull request Nov 2, 2026

vulkan : allow split_k when n is smaller than the tile width (ggml-or…

696cf56

…g#242)

Navigation Menu

Repository navigation

Implement non-greedy tokenizer that tries to maximize token lengths - #242

Implement non-greedy tokenizer that tries to maximize token lengths#242
thement merged 2 commits into
ggml-org:masterfrom
thement:thement/fix-tokenizer

thement commented Mar 17, 2023

Uh oh!

Piezoid commented Mar 17, 2023

Uh oh!

Uh oh!

Piezoid Mar 17, 2023

Uh oh!

thement Mar 17, 2023 •

edited

Loading

Uh oh!

ggerganov left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

thement commented Mar 17, 2023

Uh oh!

Piezoid commented Mar 17, 2023

Uh oh!

Uh oh!

Piezoid Mar 17, 2023

Choose a reason for hiding this comment

Uh oh!

thement Mar 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

thement Mar 17, 2023 •

edited

Loading