Repository navigation
Sync to master. - #192
Merged
Merged
Sync to master.#192
Conversation
* Don't use vdotq_s32 if it's not available `dotprod` extensions aren't available on some ARM CPUs (e.g. Raspberry Pi 4), so check for them and only use them if they're available. Reintroduces the code removed in 84d9015 if `__ARM_FEATURE_DOTPROD` isn't defined. * Update ggml.c --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Without "static" prefix, it fails to compile in clang
…th-to-ggml.py (#142) There are ways that special tokens or other new tokens could be added to the tokenizer; therefore it's probably best not to assume the vocabulary is only 32000 tokens.
* Update README.md * Update README.md remove facebook
* fixed color reset on exit * added sigint handler for ansi_color_reset * Update main.cpp --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* added ctx_size parameter * added it in more places * Apply suggestions from code review --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* add SIGINT support for _WIN32 environments * perhaps more consistent
* add ggml_rms_norm * update op num
Contributor
Author
|
Sync to master. |
phuongncn
pushed a commit
to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4
that referenced
this pull request
Apr 28, 2026
* Revert "Do not quantize activations if not necessary (ggml-org#79)" This reverts commit 0bf4d99. * Fixed compilation after revert --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
ckti
pushed a commit
to ckti/llama.cpp-consolidated
that referenced
this pull request
Jul 21, 2026
Fixes the duplicate function argument in .devops/nix/package.nix (issue ggml-org#194). Same one-line fix as ggml-org#192 (jakkunight) — closing that as a duplicate after this lands. TurboQuant: no runtime/kernel impact (Nix packaging only).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
Nathanw1014
added a commit
to Nathanw1014/llama.cpp
that referenced
this pull request
Oct 21, 2026
speculative : stop the MTP draft chain on the top-1 probability product
Nathanw1014
added a commit
to Nathanw1014/llama.cpp
that referenced
this pull request
Oct 21, 2026
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 21, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 22, 2026
…r, QSA/indexer)
old-14's qwen4exp support plus old-15's qwen4exp/QSA/HC content, kept as ONE
block because QSA is intrinsic to the qwen4exp architecture (the old-15 QSA
memory wins, HC16 and qsa3 are load-bearing for the qwen4exp graph, and the
QSA/HC op dispatch is old-15):
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the
MTP draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm
policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, lightning-indexer.cu, with the CUDA dispatch
(GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE / INDEXER_FILL) and
their CPU references; the indexer cache in llama-memory-hybrid-idx.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (+ dispatch, supports_op, the
hc_combine_mix fusion) and the HC16 planar layout.
- the managed lazy reader (llama-lazy-reader) and the loader's managed_budget.
- the GDN beta-sigmoid fusion, the DSV4_HC split states in the Meta backend,
and the QSA/HC op definitions in ggml.h / ggml.c.
Env-read hygiene (no behaviour change): resolve GGML_CUDA_GDN_CHUNKED / GGML_CUDA_GDN_CHUNKED_BF16,
GGML_CUDA_QSA_SLICES, GGML_CUDA_SPLICE_GATHER and the GDN_DBG_SKIP_* diagnostics once into values
rather than caching the getenv() pointer: the MSVC CRT returns a buffer that the next getenv() call
overwrites, so a cached pointer is unsafe on Windows.
Build fix: the GGML_CUDA_WORK_ALLOC_DEBUG read in llama_context::graph_reserve lives in src/ (not
linked against the CUDA common.cuh), so it uses a local static instead of the CUDA helper.
Folded in from four contributor PRs on v12-097cf33d9-r80 (bsp2020 / briansp2020; block 10 also owns
the rope + FWHT fusion that ggml-org#172 and ggml-org#174 put here in r78):
- fused RMS_NORM + MUL + ROPE (+ FWHT) also for M-ROPE / IMROPE (PR ggml-org#175, 0001): the matcher admits
GGML_ROPE_TYPE_MROPE / IMROPE besides NORMAL / NEOX, and the kernel computes theta with rope_multi's
own section selection and float expressions (NEOX pairing, iw == i0, VISION still rejected). This is
the TODO item recorded against ggml-org#172: every Qwen3.x model runs rope_multi, so the fusion was inert on
the dense 27B and the 35B-A3B. Kill switch GGML_CUDA_DISABLE_ROPE_MROPE_FUSE=1.
- BF16 caches off the 5-op RMS_NORM + MUL + ROPE + VIEW + SET_ROWS fusion (PR ggml-org#175, 0002): that fusion
runs ggml_cuda_op_rms_norm_mul_rope_fused, which writes F32 / F16 rows only and otherwise aborts, but
ggml_cuda_should_fuse_rope_set_rows admits a BF16 SET_ROWS. Latent on r80 for a NEOX qk-norm model
with -ctk bf16, and reachable for the Qwen3.x models once 0001 lands. The 5-op match now declines a
BF16 SET_ROWS; the RMS_NORM + MUL + ROPE fusion and the plain SET_ROWS take it instead.
- the q8_0 K and V cache writes folded into the rope + FWHT kernel (PR ggml-org#173): with a q8_0 cache the K
path ends MUL_MAT(Hadamard), RESHAPE, VIEW, SET_ROWS and the V write (VIEW, SET_ROWS) follows it, so
the fused kernel now writes both cache rows through ggml_cuda_quantize_q8_0_group (k_set_rows_q8_0_group's
body, moved into set-rows.cuh). Only the K write is folded inside a concurrent-stream region
(GGML_CUDA_GRAPH_OPT=1). Kill switch GGML_CUDA_DISABLE_ROPE_FWHT_SET_ROWS=1.
- the GDN q and k l2 norms in one launch (PR ggml-org#179): build_gdn_l2_norm normalizes both with the identical
RMS_NORM + SCALE over views of the same conv output, so rms_norm_scale_pair_f32<256> takes blocks
[0, nrows) for q and [nrows, 2*nrows) for k with the same per-row body. Matcher additions: only
no-op nodes between the two pairs, both inputs F32 views of the same view_src with equal ne / nb,
equal eps and scale params, and the second input must not read the first pair's output.
Kill switch GGML_CUDA_DISABLE_NORM_PAIR=1.
Rejected in the same review: PR ggml-org#177 (GDN direct conv also writes the recurrent snapshot tails). It is
not run-to-run deterministic on the 35B-A3B prefill under HIP graph capture and does not reproduce the
stock logits there; record in archive/work/gdn-conv-snapshot-fold/.
Prefill cache staging hook (r83): ggml_backend_cuda_stage_from_cache is live again for the expert-cache
arena. The scheduler's whole-table staging copy of a host-resident expert table (stage_upload for
-sm layer, stage_gather for the -sm tensor meta split) is filled by one kernel from the arena (resident
experts, D2D) + the pinned host master (the rest, H2D) instead of a full H2D re-upload; the destination
is the ordinary staged table in expert order, so every consumer is unchanged. It declines to the plain
upload for an unusable/stood-down table, an unpinned host master or a geometry mismatch. Kill switch
GGML_MOE_CACHE_STAGE=0.
Slab narrow band (r85): the always-resident narrow band was an estimate taken before the first ring carve
(a `split_only` reserve of the `n_rs_batch` shape, 1792 MiB on the field config), and `set_narrow_floor`
cannot correct it afterwards because the ring's VA is frozen once it holds blocks. The real MTP
decode/verify graph needs 2304 MiB, so `work_alloc` classified every decode view as wide, placed it above
the H2D staging ring and evicted 3.65 GiB/device of arena. `work_alloc` now learns the band from the real
graph: a view that is not a requested wide (prefill) pass is narrow even above the floor while no wide view
is live (the ring region is only carved while one is), and the floor + ring are re-carved from it (the ring
is idle there). With a wide view live the above-the-ring placement is kept: the ring's VA must not move.
A refused raise is reported instead of dropped silently.
Measured (2x R9700, Flash-Next IQ4_NL -ncmoe 46, 131k ctx, -b/-ub 6144, MTP n6): 18 -> 1 re-arm, decode
23.6 -> 45.4 t/s (-sm tensor) / 35.8 (-sm layer), prefill unchanged; cost: 16k prefill 1443 -> 1411 t/s
(-2.2 %) because the correctly sized band moves the ring up 512 MiB.
The "may the band move" test is `ring_armed` alone: a live WIDE view is the only thing that makes the ring
region part of the reserved layout (`ring_armed` is set exactly when the boundary covers the carved ring).
`ring_used` is a bump cursor that stays at the end after a prefill, and `boundary > narrow_floor` is true
for an unrelated narrow view one chunk above the floor; treating either as "the ring may not move" left
the band unlearned on a single device -- measured gfx1100 `-ncmoe 40`: floor 1280 MiB against a 1536 MiB
decode view, one arena eviction per prefill -> decode transition. With this test the band is learned
there too (floor 1280 -> 1536, view placed at base 0, no eviction) and the two gfx1100 greedy hashes are
unchanged.
Fold (r86): beta 12, slab-transient-absorption (CUDA/slab half). The wide-phase
auxiliary region hosts the FA staging arena, the QSA3 workspace and the chunked-GDN
scratch; the workspace pool serves its large prefill temporaries from it; a graph
capture is never started while the wide band is armed; per-device free=/free_min/
outside_max/outside_failed diagnostics; default headroom 640 + 128 x n_gpu MiB.
Fold (r91): the chunked GDN prefill writes its rollback snapshots (fixed-D sequential tail).
The chunked kernel computes no per-token states, so a partial rollback into a batch it served
restored a state it never wrote (probe: logit NMSE 0.65, maxabs 12.5; see
archive/work/rs-rollback-boundary/). A single-sequence batch above the chunked threshold now
runs the chunked kernel on its first n_tokens - D tokens and the SEQUENTIAL kernel on the last
D = max(K>16?K:16, n_rs_batch) tokens in the same launch, so snapshot slots 0..D-1 exist for
every batch. No extra memory, no extra graph, ubatch unchanged (splitting the ubatch instead
measured -13.7 % pp512 and was reverted).
The bound is structural, not observational: cparams.n_rs_batch = the widest verify batch of any
enabled speculator (common_speculative_n_max() + 1, all draft and ngram types) and is carried
into this op as param 1, so a verify batch is never chunked; the tail covers the non-speculator
partial rewind (server cancel / --cache-reuse). D is fixed, not K-dependent, so plain decode and
MTP chunk a prefill identically.
Gates (gfx1201, 35B-A3B): probe P=2050 exact 0.0 and P=2048 rounding-level (1e-3) vs 0.65
without the tail; GATED_DELTA_NET 2/2 (48/48); prefill-logit vs the arm it replaces 0.0041 (fp32)
/ 0.0054 (bf16 chunked) and vs the exact sequential reference 0.004040 / 97.550 % (fp32) /
0.004408 / 97.055 % (bf16) -- no worse than the unpatched 0.004416 / 97.318 %; plain == MTP
byte-identical (64 tok and -n 3000, all four axes); recurrent depth 130 == 130, rollback 0 FAIL;
llama-bench pp512 -0.09 %, pp2048 -0.58 %, tg128 flat. gfx1151: 0.002389 / 98.069 % vs the
replaced arm, text unchanged. Kill switch GGML_CUDA_GDN_TAIL=0.
Fold (r95): rdna3-verify-fusions (PR ggml-org#192 + PR ggml-org#193, briansp2020): the verify-band Q8_1
pre-quantize fusions (residual ADD + rms_norm -> Q8_1, rms_norm -> Q8_1, gated unary -> Q8_1,
GLU -> Q8_1) and the direct GDN conv for 2..255-token verify batches are admitted on RDNA3_0
as well. Both gates become (RDNA4 || RDNA3_0), so RDNA4 and every other backend keep the
path they had and RDNA3_5 stays out. ggml_cuda_should_fuse_mul_mat_vec_q(..., verify_band)
is the single predicate the three Q8_1 fusions share, and its kernels are row-generic and
write the values quantize_q8_1 writes, so W = 1..8 stay bit-identical. The GDN conv keeps
its pre-existing single-device restriction. GGML_CUDA_FUSE_Q8_1_VERIFY=0 /
GGML_CUDA_FUSE_GDN_CONV_VERIFY=0 still turn them off.
GDN / attention-gate decode fusions, reported by @briansp2020 (PR ggml-org#200; R9700
gfx1201). Four bit-identical fusions, each with its own kill switch:
- GGML_CUDA_FUSE_GATE_BETA_BF16 (mmvf): qwen4exp's alpha/beta BF16 projections
and softplus(alpha + dt) * a in one mul_mat_vec_f launch (alpha rows, then
beta rows; each row runs the instantiation the unfused MUL_MAT takes, so both
outputs are bit-identical). The Q8_0 pair keeps ssm_gate_beta.
- GGML_CUDA_FUSE_GDN_CONV_NORM (gdn-conv): the q/k l2 norm pair folded into the
direct conv kernel, same reduction values and lanes as
rms_norm_scale_pair_f32<256>. Taken only when both SCALE outputs are disjoint
from the conv's x/state/weight/output: the allocator may place them over the
state or x (dead after the CONCAT in graph order) which the direct kernel
still reads, and without that check 35B-A3B gave run-to-run varying output.
- GGML_CUDA_FUSE_CONT_GATE_Q8_1 (unary): the attention gate view is read in
place by the gate -> Q8_1 kernel, so the CONT launches nothing.
- GGML_CUDA_FUSE_GDN_NORM_GATE (unary): qwen4exp's GDN output RMS_NORM * w is
computed inside the gate kernel, one row per 256-thread block exactly as
rms_norm_f32<256, true> runs it.
The norm's node lives several nodes before the gate node that computes it, so
the skip and the emit are two decisions joined by a context-pending pointer.
The submitted form guarded that with two GGML_ABORTs (norm never computed / gate
node never reached). This copy adds the structural guard instead: while a norm
is pending, no other fusion may start before its gate node, and the pending
norm's own MUL (dead: the gate kernel reads xn and w directly) is skipped. The
window is measured as span 4 with a fixed signature on the only graph this fires
on (qwen4exp), so the window nodes run unfused at the cost of one plain matmul
per GDN layer, and the aborts are unreachable rather than merely unlikely.
Gates (r101 base, gfx1201, ROCBLAS_USE_HIPBLASLT=0): greedy text identical in
five arms (27B plain and MTP n3, 35B-A3B 1 GPU and 2 GPU tensor, Flash-Next
IQ3_XXS 2 GPU tensor + shared MTP); test-logits-width-probe hashes identical to
r101; prefill-logit 0.000707 / 98.755 % (the r101 numbers exactly); SSM_CONV /
RMS_NORM / SCALE / MUL_MAT oracles 0 failures; rule-5 batched decode within
noise at B=1/4/8. Decode: 27B MTP +0.2-0.3 %, Flash-Next 2 GPU tensor +0.7 %,
35B-A3B 1 GPU tg128 -0.2 % (i.e. flat; the submitter's +0.5 % did not
reproduce). No GPU faults.
GDN gate/beta BF16 fusion: multi-sequence fix + regression test (issue ggml-org#204, r108).
ggml_cuda_gate_beta_bf16_try_fuse passed the gate chain's MUL as the mmvf dst. The
launch takes its column count from dst->ne[1] and its channel count from dst->ne[2];
with more than one sequence the gate MUL is [nr, n_seq_tokens, n_seqs] while the
projections and y are flat [nr, n_tokens] / [ne10, n_tokens] with one channel, so
every channel past the first re-ran the beta rows and wrote the alpha/beta values
computed from y's first n_seq_tokens columns: every sequence but the first silently
took sequence 0's gates. The text stays coherent, which is why the same-seed and
MTP gates never saw it - only multi-sequence serving did (reported 4-stream draft
acceptance 0.85 -> 0.82 at n_seq_tokens 2).
The launch now uses the projection's own shape: both tensors are contiguous with
the same element count (checked by the matcher), so it is the same memory, written
flat. A single-sequence launch is unchanged (identical ne/nb/data), so single-stream
numerics are bit-identical; multi-sequence decode/verify now matches the unfused
chain exactly.
tests/test-fusion-gate-beta-multiseq.cpp is the regression test. It needs no model:
it builds the chain for a matrix of (n_seq_tokens, n_seqs) shapes on the GPU
backend, compares the fused launch against the same graph with the fusion made
ineligible (one extra consumer of the ADD output makes ggml_can_fuse_subgraph
reject the window), requires bit-identical gate and beta values, and proves the
fusion actually fired (the skipped SOFTPLUS output is left at a NaN sentinel), so
a change that silently stops matching fails the test instead of passing it
vacuously.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.