Improved quantize script by SuajCarrot · Pull Request #222 · ggml-org/llama.cpp

SuajCarrot · 2023-03-17T03:33:34Z

I improved the quantize script by adding error handling and allowing to select many models for quantization at once in the command line. I also converted it to Python for generalization as well as extensibility.

mattsta · 2023-03-17T17:10:08Z

there's another PR about parallelizing the quantizations. here it would be easy just to wrap this in a multiprocessing.Pool() (with the subprocess calls extracted out to a top-level function): https://docs.python.org/3/library/multiprocessing.html

I don't understand exactly what can be parallelized here, is it quantizing many models at the same time? The code from the latest commit removed the loop that acts upon the incorrect listing of the file (for i in os.listdir(model_path)) by the way.

@mattsta

Fixed and improved many things in the script based on the reviews made by @mattsta. The parallelization suggestion is still to be revised, but code for it was still added (commented).

The original Bash script uses a glob pattern to match files that have endings such as ...bin.0, ...bin.1, etc. That has been translated correctly to Python now.

sw · 2023-03-19T10:56:19Z

I think it's a good idea to remove the requirement for a Unix shell.
See also #285 which would be made obsolete by this, assuming the Python script works on Windows?

I suggest to remove quantize.sh and update the readme to use the python script.

New code to set the name of the quantize script binary depending on the platform has been added (quantize.exe if working on Windows) and the README.md file has been updated to use this script instead of the Bash one.

SuajCarrot · 2023-03-19T16:34:06Z

Thank you for your comment @sw, I just pushed a commit that applies the changes you suggested. Let me know if there's anything else that should be done to achieve full compatibility with Windows.

Fixed a typo regarding the new filenames of the quantized models and removed the shell=True parameter in the subprocess.run call as it was conflicting with the list of parameters.

This was making the automatic help message to be suggesting the program's usage as being literally "$ Quantization Script [arguments]". It should now be something like "$ python3 quantize.py [arguments]".

ggerganov · 2023-03-19T18:34:47Z

Should we merge now or wait for someone to test on Windows?

@SuajCarrot maybe keep the .sh for now and add a comment that it is deprecated. We will remove it later.

SuajCarrot · 2023-03-19T19:07:48Z

Thank you for merging! Should I create another pull request with the Bash script added back as well as its deprecation notice in the README?

ggerganov · 2023-03-19T19:09:31Z

It was confirmed in #285 that it works on Windows, so no need to do it

SuajCarrot · 2023-03-19T19:11:16Z

That's great, thank you.

* llama-eval : add per-problem summary table to HTML reports - Add chunk_idx and problem_idx to TaskState and saved case dicts - Group completed cases by problem_idx in dump_html() - Render per-problem summary table before individual task table - Columns: Problem (zero-padded), Runs, Correct (n/r), Tokens (min/avg/max), T/s (min/avg/max), Gen s (min/avg/max) - Sorted by problem index, monospace font, right-aligned numbers - Colspan headers for grouped stats, auto width - Simulator: add /v1/models endpoint, timings in response, template-aware question matching, --dataset arg (aime/aime2025) Assisted-by: llama.cpp:local pi * llama-eval : add tabs for Detailed and Summary tables, apply monospace font globally - Wrap Detailed and Summary tables in switchable tabs (Detailed active by default) - Remove summary-section wrapper, use tab labels instead - Apply monospace font to all tables and the top bar Assisted-by: llama.cpp:local pi * llama-eval : redesign top bar as CSS grid label/value pairs - Replace flat span list with 4-column grid layout (2 pairs per row) - Labels in muted color (ggml-org#888), values in dark (ggml-org#222) - Bold dataset name and model name - Removed media query, always uses 4 columns Assisted-by: llama.cpp:local pi * llama-eval : use realistic token counts and throughput in simulator - comp_tokens: [30, 80] → [10000, 60000] - tps_gen: derived → uniform [90.0, 110.0] - t_gen_ms: now computed from tokens/tps Assisted-by: llama.cpp:local pi * llama-eval : color Answer column green/red based on correctness Use the same .correct/.incorrect CSS classes on the Answer column to make correct answers green and incorrect answers red. Assisted-by: llama.cpp:local pi * llama-eval : fix pyright errors from max(..., key=len) type inference Use key=lambda x: len(x) instead of key=len so the type checker infers the return type as str instead of Sized, fixing: - unresolved-attribute: Object of type Sized has no attribute lower - not-subscriptable: Cannot subscript object of type Sized Assisted-by: llama.cpp:local pi

…rator improvements (ggml-org#24974) * Update to OV 2026.2.1, Make OV release packages self-contained * Update to OV 2026.2.1, Make OV release packages self-contained * OpenVINO Backend: Remove compute_op_type hardcoded sets (ggml-org#222) * OpenVINO Backend: Remove compute_op_type hardcoded sets * revert get_op_type removal * OpenVINO backend: enable softmax with sink input * OpenVINO backend: opt mul_mat_id convert process for large size * OpenVINO backend: Modify add_id to support 2D/4D * OpenVINO Backend: Add glu_swiglu_oai * PR review: fix paths * PR review: fix path consistency --------- Co-authored-by: Mostafa <mostafas.main.email@gmail.com> Co-authored-by: Xuejun <Xuejun.Zhai@intel.com>

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com>

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve.

…ml (ggml-org#222)

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve.

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve. Fold (r208): RDNA2 (gfx1030) FA prefill, reported and validated by @slavap (PR ggml-org#270/ggml-org#271; Radeon Pro V620 gfx1030, Qwen3.6-35B-A3B IQ4_XS, -b 8192 -ub 2048 -fa 1). This is the RDNA2 long-context deficit of issue ggml-org#214, and it is folded here rather than cut as new blocks. - q_global threshold 16 -> 64: the two kernel-side conditions, the Q_tmp_count stub and all seven launcher q_h2 arguments move together (a launcher that omits the argument is the r88 null-Q_h2 fault). At ncols 16..32 the global-half2 Q path re-reads Q per K-tile pass and the checked tail spilled ~180 VGPRs; with Q back in LDS it does not. - rdna_cap_ncols2 lifted for DKQ == 256 (D=128 keeps the cap). The cap dates from r76, when HIP's RDNA2 occupancy query returned 0 for the wide D=256 variants; launch_fattn computes RDNA2 occupancy itself now, so the cap is stale. Checked against the tree's own formula (4*waves_per_simd/waves_per_block, LDS cap 128K/lds) over the patched gfx1030 HSACO: every D=256 shape the launcher can now pick reports >= 2 blocks, so the GGML_ASSERT(max_blocks_per_sm > 0) abort the cap guarded against is unreachable. - the checked tail clamps out-of-range rows to the last in-bounds row instead of branching to a local zero array. The tail values are discarded by the k_VKQ_sup guards, every read stays inside the cache, and the pointer-select the branch existed to avoid (the kv=113 flat-scratch fault) is gone with it. i_sup - 1 may be -1 (a V chunk wholly past the KV end): row -1 is the last VALID row, and every weight in that chunk is zero, so the comment in the loader says so and warns against "fixing" it to max(i_sup - 1, 0), which would read one row past the cache. Verified here on ROCm 10.1.0 against this tree: both levers compile for gfx1030 (all 12 template-instances tile TUs, no warnings), the D=256 4x8/16x2 checked-tail scratch goes 480/500 B -> 0 B (the submitter's 488/520 on their toolchain), the largest static LDS in the whole tile family stays 63488 B of gfx1030's 65536 B, and for gfx1201 every tile kernel's machine code is bit-identical before and after (88/88 and 106/106 kernels hashed per kernel), so RDNA4, RDNA3_0, RDNA3_5, NVIDIA and MUSA/SYCL keep their codegen and numerics exactly. The timings stay the submitter's (pp2048 @ d32768 1041.0 -> 1259.8 t/s, tg256 @ d131072 50.9 -> 62.4 across their two arms); RDNA2 is best-effort per CONTAINERS.md and carries no gate in this repo. Not taken from the PRs: their own patches-rdna2-r205/ + release-rdna2-r205.json + scripts/apply-rdna2-r205.sh re-packaging (the same call as r88), and the block 16 two-lane prefill the branches were stacked on (PR ggml-org#268, a separate candidate).

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve. Fold (r208): RDNA2 (gfx1030) FA prefill, reported and validated by @slavap (PR ggml-org#270/ggml-org#271; Radeon Pro V620 gfx1030, Qwen3.6-35B-A3B IQ4_XS, -b 8192 -ub 2048 -fa 1). This is the RDNA2 long-context deficit of issue ggml-org#214, and it is folded here rather than cut as new blocks. - q_global threshold 16 -> 64: the two kernel-side conditions, the Q_tmp_count stub and all seven launcher q_h2 arguments move together (a launcher that omits the argument is the r88 null-Q_h2 fault). At ncols 16..32 the global-half2 Q path re-reads Q per K-tile pass and the checked tail spilled ~180 VGPRs; with Q back in LDS it does not. - rdna_cap_ncols2 lifted for DKQ == 256 (D=128 keeps the cap). The cap dates from r76, when HIP's RDNA2 occupancy query returned 0 for the wide D=256 variants; launch_fattn computes RDNA2 occupancy itself now, so the cap is stale. Checked against the tree's own formula (4*waves_per_simd/waves_per_block, LDS cap 128K/lds) over the patched gfx1030 HSACO: every D=256 shape the launcher can now pick reports >= 2 blocks, so the GGML_ASSERT(max_blocks_per_sm > 0) abort the cap guarded against is unreachable. - the checked tail clamps out-of-range rows to the last in-bounds row instead of branching to a local zero array. The tail values are discarded by the k_VKQ_sup guards, every read stays inside the cache, and the pointer-select the branch existed to avoid (the kv=113 flat-scratch fault) is gone with it. i_sup - 1 may be -1 (a V chunk wholly past the KV end): row -1 is the last VALID row, and every weight in that chunk is zero, so the comment in the loader says so and warns against "fixing" it to max(i_sup - 1, 0), which would read one row past the cache. Verified here on ROCm 10.1.0 against this tree: both levers compile for gfx1030 (all 12 template-instances tile TUs, no warnings), the D=256 4x8/16x2 checked-tail scratch goes 480/500 B -> 0 B (the submitter's 488/520 on their toolchain), the largest static LDS in the whole tile family stays 63488 B of gfx1030's 65536 B, and for gfx1201 every tile kernel's machine code is bit-identical before and after (88/88 and 106/106 kernels hashed per kernel), so RDNA4, RDNA3_0, RDNA3_5, NVIDIA and MUSA/SYCL keep their codegen and numerics exactly. The timings stay the submitter's (pp2048 @ d32768 1041.0 -> 1259.8 t/s, tg256 @ d131072 50.9 -> 62.4 across their two arms); RDNA2 is best-effort per CONTAINERS.md and carries no gate in this repo. Not taken from the PRs: their own patches-rdna2-r205/ + release-rdna2-r205.json + scripts/apply-rdna2-r205.sh re-packaging (the same call as r88), and the block 16 two-lane prefill the branches were stacked on (PR ggml-org#268, a separate candidate).

@slavap

… derived kq mask Band (RDNA4 WMMA GQA fold, wide verify band, GQA cap ggml-org#133, the ggml_set_fa_tensor_parallel hint), native quantized q8_0/q4_0/q4_1/q5_0/q5_1/ iq4_nl/bf16 K/V reads, the V3 derived kq-mask inputs (cell_pos/tok_lo/tok_hi), the iq4_nl FA quants and the per-type tile instances. The per-context fattn_stage prefill arena (and its RDNA3_5 prefill band split) is deferred to block 10 slab-moe: here a native-capable operand is read natively at every width and the F16 staging copy for a non-native type goes to the node's reserved scratch (the pre-arena behaviour). Env-read hygiene (no behaviour change): resolve GGML_CUDA_FA_WMMA_256 / GGML_CUDA_FA_WMMA_MAX_HEAD once into values rather than caching the getenv() pointer (unsafe on Windows). RDNA2 (gfx1030) FA gate fixes, reported and validated by @slavap (PR ggml-org#181; Radeon Pro V620 + RX 7900 XTX, ROCm 7.2.4, one gfx1030;gfx1100 build, the full FLASH_ATTN_EXT and MUL_MAT backend-ops gates green on both cards). Without them the gfx1030 FLASH_ATTN_EXT gate aborted with "ROCm error: an illegal memory access" and MUL_MAT was blocked behind it. Three separate defects: - ggml_cuda_flash_attn_ext_get_alloc_size (TILE branch): FATTN_KV_NATIVE_BF16 is the *F16 staging* type on hardware without native BF16 tensor cores -- the tile case chooser only picks the BF16 instantiation under bf16_mma_hardware_available, and gfx1030 has no V_DOT2_F32_BF16 -- so mapping it to GGML_TYPE_BF16 left need_f16 false and let the whole-cache converter write past the node's allocation (sentinel clobber, then illegal access). The mapping now matches the chooser. The same latent mismatch existed on RDNA1 and on pre-Ampere NVIDIA / older MUSA. - launch_fattn_tile_switch_ncols1: the decode/verify band (Q->ne[1] <= 8) was the only one of the seven launch sites that omitted the q_h2 argument, so it defaulted to false while a wide RDNA2 kernel (q_global: GGML_CUDA_FATTN_TILE_ Q_GLOBAL, DKQ <= 256, ncols >= 16) read Q from the Q_h2 global half2 buffer that was never allocated -- a null pointer and a fault near 0x1000 for GQA-16 decode shapes (hsk=192, nr23=[16,1], n_q <= 8). It now passes the same condition as the other six. - ggml_cuda_flash_attn_ext_get_f16_extra_data: the ncols2 > 1 tile branch loads its KV tiles without out-of-bounds checks, so the last tile reads up to nbatch_fa - 1 rows past the operand. The KV cache allocation has always had that slack, but the converted F16 scratch is exact size and sits at the end of the node's allocation, so it read into the next allocation. A 511-row guard tail is now reserved after each converted operand (the nbatch_fa config field is 9 bits and the table asserts <= 256, so half is margin), gated on GGML_CUDA_CC_IS_RDNA2 so every other backend keeps its allocation sizes. Fold (r92): the 2-byte band arm joins the 32-column band block, and the RDNA4 head-512 row goes back to upstream's shape. (a) `ggml_cuda_fattn_band_wmma_ncols1()` no longer special-cases f16/bf16 (native BF16) to 2 columns, and the 2-byte band split drops from 3*nsm/4 to nsm/2. The 2-column tile was the right default only while that arm still selected the pre-band 128-half2 config row: with the band's own 64-half2 row it wins ~5 % at n_q = 1 and loses everywhere else, because ceil(n_q/2) output tiles each stream the whole KV cache (head 256 GQA 6, gfx1201, us per call, f16 KV: n_q 1 760 -> 684, n_q 4 763 -> 695, n_q 8 1142 -> 823 at kv 98304; bf16 813/869/1451 -> 685/696/832). The choice stays width-independent, so W = 1..8 still reduce identically (GREEDY-PURITY.md §36). (b) The `(512, 512, 32)` RDNA row -- the only head-512 row the AMD dispatcher can reach, since `switch_ncols1` forces ncols = 32/ncols2 for DKQ > 256 -- was replaced by the 2026-09-14 RDNA prefill sweep with an 8-warp / 128-KV-row / nbatch_V2 16 config that is ~1.35x slower than the tile kernel, which made RDNA4's WMMA head cap of 576 a prefill regression for every Gemma 4 global layer (issue ggml-org#188). Retuned to 4 warps, 32 KV rows, one 128-half2 K/V batch, combine 64, Q in registers: gemma-4 31B global-attn shape (D=512, 4 KV heads, 32 Q heads, ub 512, q8_0 KV) 32768 126.1 -> 42.3 ms and 98304 377.0 -> 128.1 ms, end to end pp512 +95 % at d32768 and +144 % at d98304 (31B and 26B-A4B). Records: archive/work/fa-band-2byte-row/, upstream/UPSTREAM-PR-fa-512-rdna-head-cap/. Fold (r98, PR ggml-org#195, @overdoingism): with q8_0 K and V read in place (decode/verify, up to 8 query rows) the RDNA4 band launches a second kernel variant, `flash_attn_ext_f16<..., band_fetch_ahead = true>`, which fetches the raw q8_0 bytes of the next K/V batches into registers ahead of use and converts a batch into shared memory only when it is due; its barriers order shared memory only (release / s_barrier / acquire on "local"), because `__syncthreads()` also waits for every outstanding global load and would drain the fetches at the next barrier. The quants stay packed in 32-bit words, with an empty `asm volatile` pinning the conversion at the stage point (otherwise the compiler unpacks the bytes right after the fetch, which waits there and needs 4x the registers). 64-column block (widths 5..8): two batches in flight, Q in registers. 32-column block (widths 1..4): one batch in flight and the stream continues across the iteration boundary into the next KV block that is not fully masked, Q read from shared memory (a register-resident Q is spilled there and its scratch reloads share the load counter with the fetched K/V). The variant is compiled in two generated translation units of its own (`template-instances/fattn-mma-f16-band-fetch-ahead-ncols1_{4,8}- ncols2_8-dkq256-dv256.cu`, emitted by generate_cu_files.py), so every other launch, including f16 / bf16 K/V and prefill, compiles to the same gfx1201 instructions as before it (verified at the assembly level: 0 differing instructions in both dkq256 band instances, a dkq128 instance, the tile and vec units and fattn.cu). Default on; `GGML_HIP_FA_BAND_FETCH_AHEAD=0` runs the kernel without it for q8_0 too. It also keys the FA shared-memory attribute raise per (variant, device) rather than per device, so the two kernels cannot share one "already raised" flag. Gates (gfx1201, ROCm 10.1.0, 27B UD-Q4_K_XL with q8_0 K/V unless noted): bit-identical to r97 everywhere the variant fires -- `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 on both builds, prefill-logit KLD 0.000000 / same-top-p 100.000 %, greedy text identical at 8K and 110K context, on a 2 GPU tensor split and on gemma4 12B (its head-256 SWA layers), the 4-axis adaptive MTP gate at -n 3000 (text and draft acceptance identical on all four axes) and n7 = plain; `llama-batched-bench -npp 16 -ntg 32 -npl 1,4,8` B=1 29.47 -> 30.12 t/s, B=4 95.73 -> 97.70, B=8 flat; decode +0.7 % at 8K and +2.4 % at 110K context (3 interleaved pairs each; prefill unchanged at 840 t/s), +0.9 % on the 2 GPU tensor split. Fold (r99, PR ggml-org#198, @slavap): two RDNA2 (gfx1030) FA follow-ups from the review of PR ggml-org#181, plus the arch-independent half they needed. - **One tile K/V type selection.** The tile kernel has a single K/V type parameter and its choice was kept by hand in two places: the instantiation (`ggml_cuda_flash_attn_ext_tile_case_type`, `fattn-tile.cu`) and the F16 scratch sizing (`ggml_cuda_flash_attn_ext_get_alloc_size`, `fattn.cu`). They had drifted three times (the missing q4_0 arm, the mixed-K/V NaNs, the missing RDNA2 bf16 scratch). Both now call one `ggml_cuda_fattn_tile_type(K, V, cc)` in `fattn-rdna.cuh`: BF16 only where the hardware reads it natively, else the native dequant type, else F16 (BF16 without native support maps to F16, i.e. the launcher converts). The mapping is 1:1 with the previous order on every arch, the instantiation set is unchanged (96 tile objects before and after) and the host change leaves every tile kernel's gfx1201 ISA identical. - **Checked tail for the last unchecked tile (RDNA2 only).** The `ncols2 > 1` tile branch loads its KV tiles without out-of-bounds checks, so its last tile can read up to `nbatch_fa - 1` rows past the KV data. In a model that stays inside the KV cache tensor's slack, but an exact-size operand (test-backend-ops) can reach the neighbouring allocation. On RDNA2 the last tile now takes the already-compiled `oob_check = true` form (zero-fills the tail and keeps it out of the softmax). Every other target is untouched: `RDNA2` is defined only in the gfx103x device pass (`vendors/hip.h`), the added block is inside `#if defined(RDNA2)`, and the gfx1201 tile instances are byte-identical in ISA to r98. - **The 511-row guard tail is kept** and its derivation documented: `nbatch_fa` is a 9-bit config field but `GGML_CUDA_FATTN_TILE_CONFIG_CASE` static_asserts `<= 256`, each K chunk loads `nbatch_fa` rows and each V chunk `nbatch_V` with `static_assert(nbatch_fa % nbatch_V == 0)`, so the over-read is at most `255` rows; 511 = 2 x 256 - 1. Since the RDNA2 tail is now checked, the tail is belt-and-braces, not the only protection. Comment only: no allocation size changes on any backend. Gates (gfx1201, ROCm 10.1.0): `test-backend-ops -o FLASH_ATTN_EXT` 6391/6391 (identical per case to r98), prefill-logit KLD 0.000000 / 100.000 %, greedy text identical to r98, `validate-set.sh` green, per-block build green. A `gfx1201;gfx1030` dual-target build compiles the RDNA2 arm; the RDNA2 runtime gate is the reporter's (Radeon Pro V620, ROCm 7.2.4: FLASH_ATTN_EXT 6389/6389, MUL_MAT 2444/2444 with the add-on, and the same on gfx1100). RDNA3_0 (gfx1100) head-512 WMMA prefill rows, reported by @briansp2020 (PR ggml-org#199; RX 7900 XTX). The RDNA head-512 rows were tuned on gfx1201 and keep Q in registers, so on gfx1100 every head-512 kernel sat at the 256-VGPR limit and spilled (~1.0-1.4k VGPRs per lane); head 512 was therefore capped back to the tile kernel on RDNA3_0 in r5 (issue ggml-org#30). RDNA3_0 now has its own rows for DKQ == DV == 512 (Q in shared memory, 128 threads, 64-wide K/V/combine batches), and head 512 takes WMMA prefill again on top of the 256 cap: other heads above 256 keep the tile kernel, RDNA4/RDNA3_5 rows are unchanged and the decode/verify band stays on the tile kernel. GGML_CUDA_FA_RDNA3_WMMA_512=0 disables it; GGML_CUDA_FA_WMMA_256=0 and GGML_CUDA_FA_WMMA_MAX_HEAD still take it out (the rows are RDNA3_0-only, so MAX_HEAD only decides whether head 512 takes WMMA at all). Measured on gfx1100 (gemma-4-26B-A4B QAT UD-Q4_K_XL, q8_0 KV, pp2048): d32768 1908.3/1909.2 -> 2549.1/2554.6 t/s (+34 %), d65536 1209.3/1207.6 -> 1756.2/1757.5 (+45 %), tg128 at d32768 unchanged (127.74/127.74 -> 127.80/127.73). FLASH_ATTN_EXT 6391/6391 on gfx1100; the switch-off text is bit-identical to r101 and gfx1201/gfx1151 output is unchanged (byte-identical on gemma4-26B-A4B). Prefill-logit NOTE (known limitation, TODO ggml-org#81 / wip/fa-rdna3-head512-purity): this is a tile->WMMA family switch, so it moves the prefill logits against the tile kernel: mean KLD 0.0293 / same-top-p 92.59 % on that model at the gate's out-of-domain point (PPL 604), i.e. it does NOT meet the absolute 0.005 / 98 % gate. Calibrated at the same point: the head-256 WMMA rows the delivery has shipped default-on for gfx1100 since r5 measure 0.0441 / 90.63 %, a benign ubatch change on the same kernel ~0.12 per chunk, and the issue-ggml-org#113 BF16 chunked GDN 0.62. PPL is unchanged within its +/-5 % error bar (f16 KV 604.06 vs 607.20, q8_0 KV 617.85 vs 620.59). Accepted as no worse than the shipped equivalent switch; the campaign above exists to close the gap. kq-mask fill guards, reported, root-caused and validated by @RikshaDriver (PR ggml-org#222, issue ggml-org#221; RX 7600 gfx1102, ROCm 10.1, bailingmoe3 / Ling-3.0-tiny-IQ4_NL). Only a single-sequence context enables the derived kq form, so the packed mask in llm_graph_input_mem_hybrid_k is left with no consumer, the allocator gives an unconsumed input no buffer, and the unconditional fill asserted on it (GGML_ASSERT(buffer) via llama_kv_cache::set_input_kq_mask) - llama-cli aborted before the first token. Ten of the thirteen fill sites in src/llama-graph.cpp already guard for exactly that; llm_graph_input_mem_hybrid_k and llm_graph_input_attn_k_dsa (its mla and lid masks) now carry the same guard as their siblings. can_reuse needs nothing: can_reuse_kq_mask() reads ne[] only. The decode/verify band's fully-masked KV group skip, reported by @overdoingism (PR ggml-org#223, the same reporter whose issue ggml-org#45 the band came from). The prefill skip built its packed bitmap under Q->ne[1] > 8, so the band read every other slot's -INF cells on every decode step - the cost LLAMA_SERVER_LAZY_IDLE_SLOTS (r130) made common. The band now gets the bitmap too and its loop, which already handled it (and had never been reached), skips the fully masked iterations while keeping its round-robin split: block i still owns i, i+P, ..., so the accumulation order over the rest is unchanged and the result is bit-identical. GGML_CUDA_FA_BAND_SKIP=0 (band arm) / GGML_CUDA_FA_MASK_SKIP=0 (the whole mechanism) turn it off. Measured gfx1201, 27B UD-Q4_K_XL, 2 slots unified, q8_0 KV: a resident ~105K conversation behind an active 9.8K one 50.84 -> 41.69 ms/step (-18 %, token ids identical; with a DFlash2 draft at n-max 5: 18.62 -> 14.83), -0.6 % with nothing to skip. The delivery adds the decode-width packed-hole op cases (the band skip had no oracle) and evaluates the band predicate once for both the skip gate and the 2-D launch. Co-authored-by: overdoingism <overdoingism@users.noreply.github.com> Fold (r134, issue ggml-org#224): the band's 64-column block (query widths 5..8) gets its own geometry and its own instantiation. A band block's per-column reduction splits the KV rows of a tile nwarps*cols_per_warp/ncols (np) ways, and that split is part of the width purity contract: the 32-column block (widths 1..4, 8 warps, np 4) and the 64-column block must split them identically, or greedy decode with a draft differs from decode without one. r126 removed the 16-warp row for ncols 64 (it was also the prefill instantiation), which left widths 5..8 on prefill's 8-warp table row (np 2) -- a reordered reduction, ~1e-3 relative, invisible to every throughput number, same-seed text gate and CPU-referenced op test, and reported as an 8K/110K greedy divergence on dense head-256 GQA-6 (Qwen3.8-27B + DFlash2, plain != draft on r131, identical with GGML_HIP_FA_BAND_WIDE=0 and on r123). The 16-warp band row is back (fattn_mma_config(512, 1, 64, 64, 64, 32, 1, true), nbatch_combine 32 to stay inside the 64 KiB LDS limit at 16 warps), selected by a new compile-time `band_wide` flag threaded through the config getters, the case function and the kernel, so prefill keeps the 256-thread table row and its deep-prefill win (pp512 @ d98304 unchanged). The band's 64-column case + its q8_0 fetch-ahead variant are separate translation units (template-instances/fattn-mma-f16-band-wide-*.cu, emitted by generate_cu_files.py). A static_assert on the two band rows (np and the KV batch shape must match) makes this class of regression a build error. Fold (r164, PR ggml-org#242, overdoingism): `llm_graph_input_kq_derived::can_reuse()` looks through a `llama_memory_hybrid_context` to its attention cache. A hybrid model's `params.mctx` is the wrapper, never the `llama_kv_cache_context` the input was built from, so the derived input rejected reuse for *every* single-sequence batch of more than 8 tokens - and since `LLAMA_KV_N_PAD_MIN` holds `n_kv` on the 256-cell stride the derived form's own gate requires, those batches really did take the derived form, so each one rebuilt its graph: a 9-16 row n-gram verification step, or an MTP verify step at `--spec-draft-n-max >= 8` (0 graphs reused over 300 decode tokens at a fixed depth 15, i.e. 6820 -> 6338 ms on 3 GPU `-sm tensor` and 5329 -> 5041 ms at 2 GPU, +7.6 % / +5.7 % - against this block's own `LLAMA_KQ_MASK_DERIVED=0`). `get_attn()` is exactly the cache `build_inp_mem_hybrid()` built the input from, and the existing checks (n_kv, stream count, token count, `kq_mask_derivable()`) then run against it unchanged, so reuse returns to parity with the packed path (n-gram n_max 15 on 1 GPU: 58 -> 77 reused = the packed arm; 3 GPU: 0 -> 97) with greedy text and draft acceptance byte-identical on every arm. iSWA still rejects, necessarily: `build_inp_mem_hybrid_iswa()` creates *two* derived inputs (one bound to the base cache, one to the SWA cache), so there is no single cache to look through. `LLAMA_KQ_MASK_DERIVED_REUSE=0` restores the rejection. Fold (r172, PR ggml-org#247, overdoingism): the derived kq mask covers batches of several sequences, so a `--parallel > 1 --kv-unified` context stops reserving the packed mask (n_kv x n_ubatch x 2 bytes on each device and again pinned on the host). Sequence k of the ubatch has its cells and its tokens' [lo, hi] shifted by k * 2^23 (KQ_DERIVED_SEQ_STRIDE), so the existing `tok_lo <= cell_pos <= tok_hi` predicate keeps every token inside its own sequence's range and the FA kernels need no change. The form is taken only when one KV stream holds the batch, every token is in exactly one sequence, no cell carries two of the batch's sequences (`llama_kv_cells` now counts those, so the check is O(1) unless `llama_memory_seq_cp` was used) and every position is below the stride; a multi-sequence batch of at most 32 tokens (the multi-slot verify steps) and every other refused shape keeps the packed mask. `kq_mask_packed_reachable()` drops its `n_seq_max > 1` term, and `decode()` latches `kq_mask_multiseq_seen` on the first pre-split batch the form cannot serve and re-reserves once with the packed mask (the r160 2-D latch pattern, so no later batch grows the compute buffer - issue ggml-org#42). Measured (gfx1201, dense 27B UD-Q4_K_XL, `-c 245760 -ub 2048 --parallel 2 --kv-unified -ctk/-ctv q8_0`): device memory 27.51 -> 26.41 GB and server RSS 1.97 -> 0.98 GiB after load, two concurrent 16K prefills unchanged (26.96 vs 27.23 s at 1 GPU; 17.07 vs 17.09 s at 2 GPU `-sm tensor`), single-sequence greedy text byte-identical (dense / 35B-A3B MoE / hybrid 4B) and `plain == draft-mtp` identical. Purity: the derived predicate *is* the packed one - the author's host oracle compares the two fills cell by cell, re-run here with 0 mismatches on llama-perplexity at 4 sequences per ubatch (dense 27B at 1 and 2 GPU tensor, Qwen3.5-4B hybrid, Gemma4-12B and gpt-oss-20b iSWA) and on the 2-slot server shape; the prefill-logit gate at `--ub 2048 --ab "LLAMA_KQ_MASK_DERIVED_MULTISEQ=0"` reads 0.000000 / 100.000 % at both f16 and q8_0 KV, and `gate-park-idle-slots.sh` passes at 3 GPU tensor. No effect on a graph that also holds an `allow_derived = false` builder (qwen4exp/QSA, MLA): its packed mask is created and filled anyway - measured identical VRAM and RSS on Flash-Next. `LLAMA_KQ_MASK_DERIVED_MULTISEQ=0` restores the single-sequence rule and the packed reserve. Fold (r208): RDNA2 (gfx1030) FA prefill, reported and validated by @slavap (PR ggml-org#270/ggml-org#271; Radeon Pro V620 gfx1030, Qwen3.6-35B-A3B IQ4_XS, -b 8192 -ub 2048 -fa 1). This is the RDNA2 long-context deficit of issue ggml-org#214, and it is folded here rather than cut as new blocks. - q_global threshold 16 -> 64: the two kernel-side conditions, the Q_tmp_count stub and all seven launcher q_h2 arguments move together (a launcher that omits the argument is the r88 null-Q_h2 fault). At ncols 16..32 the global-half2 Q path re-reads Q per K-tile pass and the checked tail spilled ~180 VGPRs; with Q back in LDS it does not. - rdna_cap_ncols2 lifted for DKQ == 256 (D=128 keeps the cap). The cap dates from r76, when HIP's RDNA2 occupancy query returned 0 for the wide D=256 variants; launch_fattn computes RDNA2 occupancy itself now, so the cap is stale. Checked against the tree's own formula (4*waves_per_simd/waves_per_block, LDS cap 128K/lds) over the patched gfx1030 HSACO: every D=256 shape the launcher can now pick reports >= 2 blocks, so the GGML_ASSERT(max_blocks_per_sm > 0) abort the cap guarded against is unreachable. - the checked tail clamps out-of-range rows to the last in-bounds row instead of branching to a local zero array. The tail values are discarded by the k_VKQ_sup guards, every read stays inside the cache, and the pointer-select the branch existed to avoid (the kv=113 flat-scratch fault) is gone with it. i_sup - 1 may be -1 (a V chunk wholly past the KV end): row -1 is the last VALID row, and every weight in that chunk is zero, so the comment in the loader says so and warns against "fixing" it to max(i_sup - 1, 0), which would read one row past the cache. Verified here on ROCm 10.1.0 against this tree: both levers compile for gfx1030 (all 12 template-instances tile TUs, no warnings), the D=256 4x8/16x2 checked-tail scratch goes 480/500 B -> 0 B (the submitter's 488/520 on their toolchain), the largest static LDS in the whole tile family stays 63488 B of gfx1030's 65536 B, and for gfx1201 every tile kernel's machine code is bit-identical before and after (88/88 and 106/106 kernels hashed per kernel), so RDNA4, RDNA3_0, RDNA3_5, NVIDIA and MUSA/SYCL keep their codegen and numerics exactly. The timings stay the submitter's (pp2048 @ d32768 1041.0 -> 1259.8 t/s, tg256 @ d131072 50.9 -> 62.4 across their two arms); RDNA2 is best-effort per CONTAINERS.md and carries no gate in this repo. Not taken from the PRs: their own patches-rdna2-r205/ + release-rdna2-r205.json + scripts/apply-rdna2-r205.sh re-packaging (the same call as r88), and the block 16 two-lane prefill the branches were stacked on (PR ggml-org#268, a separate candidate).

Improved quantize script

f8db3d6

I improved the quantize script by adding error handling and allowing to select many models for quantization at once in the command line. I also converted it to Python for generalization as well as extensibility.

gjmulder added the enhancement New feature or request label Mar 17, 2023

mattsta reviewed Mar 17, 2023

View reviewed changes

Comment thread quantize.py Resolved

mattsta reviewed Mar 17, 2023

View reviewed changes

Comment thread quantize.py Resolved

mattsta reviewed Mar 17, 2023

View reviewed changes

Comment thread quantize.py Resolved

mattsta reviewed Mar 17, 2023

View reviewed changes

Comment thread quantize.py Resolved

mattsta reviewed Mar 17, 2023

View reviewed changes

SuajCarrot added 3 commits March 18, 2023 21:36

Fixes and improvements based on Matt's observations

2ab3311

Fixed and improved many things in the script based on the reviews made by @mattsta. The parallelization suggestion is still to be revised, but code for it was still added (commented).

Small fixes to the previous commit

01237dd

Corrected to use the original glob pattern

c028226

The original Bash script uses a glob pattern to match files that have endings such as ...bin.0, ...bin.1, etc. That has been translated correctly to Python now.

Added support for Windows and updated README to use this script

e2bfaeb

New code to set the name of the quantize script binary depending on the platform has been added (quantize.exe if working on Windows) and the README.md file has been updated to use this script instead of the Bash one.

sw reviewed Mar 19, 2023

View reviewed changes

Comment thread quantize.py Resolved

Merge remote-tracking branch 'upstream/master' into suajcarrot-changes

5d864c1

sw reviewed Mar 19, 2023

View reviewed changes

Comment thread quantize.py Resolved

SuajCarrot added 3 commits March 19, 2023 12:15

Fixed a typo and removed shell=True in the subprocess.run call

b802b78

Fixed a typo regarding the new filenames of the quantized models and removed the shell=True parameter in the subprocess.run call as it was conflicting with the list of parameters.

Corrected previous commit

c389c69

Small tweak: changed the name of the program in argparse

e9c3343

This was making the automatic help message to be suggesting the program's usage as being literally "$ Quantization Script [arguments]". It should now be something like "$ python3 quantize.py [arguments]".

sw approved these changes Mar 19, 2023

View reviewed changes

sw mentioned this pull request Mar 19, 2023

Fix scripts to support cross-platform execution #285

Closed

ggerganov approved these changes Mar 19, 2023

View reviewed changes

ggerganov merged commit 7392f1c into ggml-org:master Mar 19, 2023

This was referenced Mar 19, 2023

Python3 script instead of bash #184

Closed

Parallel Quantize.sh, add & #106

Closed

Bearsaerker mentioned this pull request Mar 12, 2025

Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352

Closed

Nathanw1014 added a commit to Nathanw1014/llama.cpp that referenced this pull request Oct 31, 2026

ci : release Windows and Vulkan zips, pin ROCm 10.1, retire release.y…

c8912d7

…ml (ggml-org#222)

Navigation Menu

Repository navigation

Improved quantize script - #222

Improved quantize script#222
ggerganov merged 9 commits into
ggml-org:masterfrom
SuajCarrot:suajcarrot-changes

SuajCarrot commented Mar 17, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mattsta Mar 17, 2023

Uh oh!

SuajCarrot Mar 19, 2023

Uh oh!

sw commented Mar 19, 2023

Uh oh!

SuajCarrot commented Mar 19, 2023

Uh oh!

Uh oh!

Uh oh!

ggerganov commented Mar 19, 2023

Uh oh!

SuajCarrot commented Mar 19, 2023

Uh oh!

ggerganov commented Mar 19, 2023

Uh oh!

SuajCarrot commented Mar 19, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

SuajCarrot commented Mar 17, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mattsta Mar 17, 2023

Choose a reason for hiding this comment

Uh oh!

SuajCarrot Mar 19, 2023

Choose a reason for hiding this comment

Uh oh!

sw commented Mar 19, 2023

Uh oh!

SuajCarrot commented Mar 19, 2023

Uh oh!

Uh oh!

Uh oh!

ggerganov commented Mar 19, 2023

Uh oh!

SuajCarrot commented Mar 19, 2023

Uh oh!

ggerganov commented Mar 19, 2023

Uh oh!

SuajCarrot commented Mar 19, 2023

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants