Repository navigation
qwen: merge tensors on convert - #31028
Merged
Merged
Conversation
CISC
reviewed
Oct 23, 2026
Comment on lines
+231
to
+246
| // matmuls that read the same input: try the merged [a | b] weight the converter writes, else the separate ones | ||
| auto create_merged = [&](llm_tensor t_merged, llm_tensor t_a, llm_tensor t_b, int64_t n_in, int64_t n_a, int64_t n_b, | ||
| ggml_tensor *& merged, ggml_tensor *& a, ggml_tensor *& b) { | ||
| if (flags & TENSOR_SKIP) { | ||
| const int skip = TENSOR_NOT_REQUIRED | TENSOR_SKIP; | ||
| create_tensor(tn(t_merged, "weight", il), { n_in, n_a + n_b }, skip | TENSOR_SKIP_IF_VIRTUAL); | ||
| create_tensor(tn(t_a, "weight", il), { n_in, n_a }, skip); | ||
| create_tensor(tn(t_b, "weight", il), { n_in, n_b }, skip); | ||
| return; | ||
| } | ||
| merged = create_tensor(tn(t_merged, "weight", il), { n_in, n_a + n_b }, flags | TENSOR_NOT_REQUIRED | TENSOR_SKIP_IF_VIRTUAL); | ||
| if (!merged) { | ||
| a = create_tensor(tn(t_a, "weight", il), { n_in, n_a }, flags); | ||
| b = create_tensor(tn(t_b, "weight", il), { n_in, n_b }, flags); | ||
| } | ||
| }; |
Member
There was a problem hiding this comment.
Would this be a good time to just introduce OTF merging?
Contributor
Author
There was a problem hiding this comment.
OTF is harder problem because that involves changing llama-model-loader and last time I did it seems to break mmap and a bunch of other things.
CISC
approved these changes
Oct 23, 2026
am17an
added
the
merge ready
Member
|
Is the merge done unconditionally or do we need to pass some extra flag to the convert script? |
Contributor
Author
|
It's done unconditionally. I don't see a reason to not merge them |
ggerganov
reviewed
Oct 25, 2026
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 26, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
edwardyoon
added a commit
to edwardyoon/focus-llama
that referenced
this pull request
Oct 26, 2026
3 of 6 tasks
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 26, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 26, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 26, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 27, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 31, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 31, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 31, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 31, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 1, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 1, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 1, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 1, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 1, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 1, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 1, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 1, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 2, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 2, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 3, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 3, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 4, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 4, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 5, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 5, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 5, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 5, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 6, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 6, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 7, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 8, 2026
…r, QSA/indexer)
The qwen4exp (Qwen3.8-Flash-Next) architecture support, split out of the old
block 10 (r109) so the model graph, the sparse-attention op and the
hyper-connections no longer share a commit with the slab, the fusion dispatcher
and the async input ring. The tree is unchanged: block 10 plus this commit is
exactly what the old block 10 carried.
Contents:
- src/models/qwen4exp.cpp: the model, the HC (hyper-connection) graph, the MTP
draft head, the PLE gather, the QSA selection/attention graph
(build_qsa_store_k / build_qsa_top_k / build_attn_qsa) and the QSA arm policy.
- QSA/indexer: fattn-qsa.{cu,cuh}, fattn-qsa3.cu, indexer-score.{cu,cuh},
indexer-topk.{cu,cuh}, the 4-head WMMA indexer scorer in lightning-indexer.cu,
with the CUDA dispatch (GGML_OP_FLASH_ATTN_QSA / INDEXER_TOPK / INDEXER_SCORE /
INDEXER_FILL) and their CPU references; the keys-only indexer cache
(LLAMA_QSA_KEYS_ONLY, no V tensor at all) in llama-memory-hybrid-idx and the
v_enabled parameter on llama_kv_cache.
- HC ops GGML_OP_HC_MIX / HC_COMBINE (dispatch, supports_op, the Meta split
states, the hc_combine_mix / hc_mix / hc_combine_norm / hc_gate_mix fusion
arms and the HC16 planar layout) in hc-mix.{cu,cuh} and hyperconn.{cu,cuh}.
- The managed lazy reader (llama-lazy-reader) and the loader's managed_budget
(LLAMA_LAZY_BUF_MB, TENSOR_SKIP_MANAGED, lazy_read::add_range) for the PLE
n-gram table; off by default (it measured slower than both the mmap and the
resident path on gfx1151).
- The QSA/HC op definitions in ggml.h / ggml.c, the scheduler op allowlists, the
CPU references, and the qwen4exp tensor-split arch gate in llama-arch.cpp (with
the ggml-hip CMakeLists publishing GGML_USE_HIP to the main libraries so the
dynamic-backend build reads it).
- The qwen4exp MTP draft context gets a hybrid-idx memory instead of a plain KV
cache (LLAMA_MTP_SPARSE=0 restores the dense draft), and llama_memory_recurrent
stops warning about the speculative verify positions for a context whose
recurrent filter matches nothing.
- Loader/user-path fix: TENSOR_READ_LAZY is only consulted when the loader has
files (the llama_model_init_from_user path has nothing to lazy-read from).
- The qwen4exp-family fusion arms that live in this block's own TUs, and the
graph-optimizer passes they need: the HC_BLK16/RES16 BF16-stream marks and
their alloc deps, and the prefill indexer relu-sum alloc dep.
- Tests: test-hc-mix and test_flash_attn_qsa in test-backend-ops (the QSA CPU
oracle: `test-backend-ops -o FLASH_ATTN_QSA`, 18/18 minimum), test-lazy-reader,
and the lazy_buf_size handover in test-llama-archs.
Split note (r109): the infrastructure this block is built on -- the slab, the
fusion dispatcher TU, the async input ring, the GDN/MMB/HC16 kernel work -- is
block 10 and is listed there. A qwen4exp-only change that is implemented inside
a general kernel TU (the GDN gate/beta BF16 fusion, the HC16 MMB BF16 streams,
the weighted-down reduction, the PLE conv fusion) deliberately stays with that
TU in block 10; only the code that needs this block's own symbols is here.
- The separate-tensor (pre-ggml-org#31028) HC mixer emits its inject product right
behind the down product, i.e. adjacent in the graph, and expands both: the two
BF16 GEMMs read the same F32 `xn`, and the CUDA BF16 src1 cache (block 10)
only holds one converted tensor. ggml-org#31028's ordering (inject last, after the
gate product whose src1 is the silu output, which evicts the entry) converted
the whole [hc*n_embd, n_tok] tensor twice per mixer -- measured on Flash-Next
GSQ-IQ3_XXS as 2x the F32->BF16 `convert_unary_cont_vec4` launches and +3.4 %
prefill when restored (issue ggml-org#226 follow-up).
R175 amendment: the QSA combine launch passes the new `block_q8_1 * q8` argument
of `flash_attn_combine_results` (added by block 10's FA-combine Q8_1 publish) as
`nullptr` -- QSA's output is not fed to a quantized matmul, so it publishes
nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
There are a few tensors in the qwen graph which share the same activations, this PR merges their weight on convert (similar to
--fuse-gate-up-exps) but for these smaller tensors. Using the new GGUF is +7% TG and +5% prefill for me on my setup.Additional information
Requirements