model : add Qwen4-Exp by JJJYmmm · Pull Request #27739 · ggml-org/llama.cpp

JJJYmmm · 2026-08-26T13:28:14Z

Overview

add Qwen4Exp. Ref: huggingface/transformers#48337

conversion example:

# backbone
python3 convert_hf_to_gguf.py --outtype q8_0 --outfile qwen4exp-Q8_0.gguf /path/to/Qwen4-Exp

# vision encoder
python3 convert_hf_to_gguf.py --mmproj --outtype f16 --outfile mmproj-qwen4exp-F16.gguf /path/to/Qwen4-Exp

# mtp
python3 convert_hf_to_gguf.py --mtp --outtype q8_0 --outfile mtp-qwen4exp-Q8_0.gguf /path/to/Qwen4-Exp

run:

llama-server -m qwen4exp-Q8_0.gguf --mmproj mmproj-qwen4exp-F16.gguf \
    -md mtp-qwen4exp-Q8_0.gguf -c 262144 -np 2 -ngl 99 -ot "ple_ngram_embd=CPU"

-ot "ple_ngram_embd=CPU" keeps the n-gram table in host memory. It is large and each token
reads only a few of its rows.

Not supported

currently mtp head only support dense attention, it should use qsa as well, but I test it and found the thourghput was no better, so in this pr I revert it for simplicity.

This PR was Assisted by Qwen3.8 MAX

JJJYmmm · 2026-08-26T13:40:40Z

This pr was primarily generated by qwen3.8max, and my role was mainly to verify its correctness. I'll try to clean it up now 🫡

danielhanchen · 2026-08-26T13:41:23Z

Oh haha I was just about to open a PR as well - see #27742

JJJYmmm · 2026-08-26T13:55:00Z

@danielhanchen sorry daniel, I forgot unsloth 😭😭 I got too excited about day0 supporting our next generation model and totally forgot 😭 considering that you’ll put your gguf on hf and people can enjoy it asap, I think I can close this pr and you can open a new one. I’ll do the numeric correctness check on my side.

JJJYmmm · 2026-08-26T13:56:09Z

let‘s move to #27742 ❤️

danielhanchen · 2026-08-26T13:58:17Z

Oh no no haha it's fine - tbh there are some very useful things from urs haha - I'm already seeing if we can reconcile

CISC · 2026-08-26T13:57:33Z

        SINKHORN_ITERATIONS  = "{arch}.hyper_connection.sinkhorn_iterations"
        EPSILON              = "{arch}.hyper_connection.epsilon"

+    class Ple:


Suggested change

class Ple:

class PerLayerEmbedding:

CISC · 2026-08-26T14:12:45Z

+
+    // the MTP block is one sparse-attention layer with its own input fusion. It has no PLE and no
+    // linear attention, so it is built here instead of in the loop above.
+    for (int il = n_layer; il < n_layer + (int) hparams.n_layer_nextn; ++il) {


Suggested change

for (int il = n_layer; il < n_layer + (int) hparams.n_layer_nextn; ++il) {

for (int il = n_layer; il < n_layer_all; ++il) {

Upstream ggml-org#27742 (the qwen4exp PR we merged) deliberately drops the MTP block: conversion/qwen4exp.py carried supports_mtp_export = False / no_mtp = True with the note "the MTP block is a separate draft head; vLLM drops it too". The effect is that EVERY GGUF built from that converter silently lacks the head, Unsloth's UD-* quants included, and the tensors cannot be recovered from a quantized file afterwards -- so speculative decode was unavailable to anyone using them. The weights do exist upstream: Qwen/Qwen3.8-Flash-Next ships 31 mtp.* tensors and text_config.mtp_num_hidden_layers = 1. A second, independent implementation of this arch -- ggml-org#27739 by JJJYmmm -- had full MTP support and was closed for logistics, not on technical grounds (its author deferred to the Unsloth PR so the GGUFs would reach HF sooner). This ports its MTP block onto our ggml-org#27742 backbone. Fetched branch pinned locally as pr-27739-qwen4exp @ dfa0c0f. The two implementations decompose the arch differently, so graph_mtp is re-expressed against this fork's builders rather than lifted: their build_hc_pre/build_hc_post are our build_hc_mix/build_hc_combine (identical signatures), and their hparams.hc_count is our hparams.dsv4_hc_mult. Everything else already existed from the fork's draft-mtp work -- llm_graph_input_embd_h, t_h_nextn, LLM_GRAPH_TYPE_DECODER_MTP, n_layer_nextn, load_mtp. What the block is: ONE dense-attention layer predicting the next-next token from the target's hyper-connection streams plus the embedding of the token the target just produced. It carries its own 512-expert MoE, so the draft is ~4B and itself sparse -- not a cheap dense head. Attention runs dense on purpose: ggml-org#27739 measured a QSA draft as no faster, and a QSA indexer would need its own cache. Notable details, none of them obvious: - hnorm is hc_dim wide, not n_embd: the reference norms the whole hyper-connection row and only then splits it into streams (deepseek4 norms each stream instead). - eh_proj is the concatenated projection. The checkpoint has two matrices, mtp.fc_embedding and mtp.fc_hidden, and A*e + B*h == [A|B]*concat(e, h), so generate_extra_tensors joins them into the single [2*n_embd, n_embd] tensor the tensor map already knows. Exact, not an approximation. - The MTP block brings its own hyper_connection_mixer, which takes the model-level slot in an MTP-only file (this arch has no output_norm -- the final mixer carries it), so filter_tensors renames it before _QwenMtpMixin drops it as a non-MTP tensor. - compress_ratios must now be block_count long, not n_layer: the loader reads it at n_layer_all, which includes the MTP block, and llama_model_loader rejects a short array outright ("expected 49, got 48"). The MTP layer gets 0, which is how that array spells "not a QSA layer" -- correct, since its attention is dense. A --no-mtp conversion is unchanged. Verified end to end on the real checkpoint: pulled the 31 mtp.* tensors plus embed/lm_head/hc-mixer by byte range from HF (6.49 GB of the 360 GB repo, via safetensors data_offsets), converted with --mtp, and quantized to Q8_0 -- 7400.33 MiB -> 3935.32 MiB, 8.51 BPW. blk.48.nextn.eh_proj lands [5120, 2560] (= 2*n_embd, confirming the concat) and blk.48.nextn.hnorm [10240] (= hc*n_embd, confirming the row-wide norm). Q8 rather than Q4 is deliberate for a draft head: Colibri measured int4 MTP collapsing draft acceptance to 0-4% against 39-59% at int8. Not yet run: the draft has not been exercised against a target. Speculative decode with it is untested. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NjfM4FxGPEhoApv6yvWAzD

Ports the multi-token-prediction draft block (one full qwen4exp layer, dense attention, own 512-expert MoE) onto the current qwen4exp tree, with mtp-only sidecar detection. Reference implementation: ggml-org#27739 (Qwen team). Works with --spec-type draft-mtp[,ngram-mod]; measured +50% code decode on gfx1151 with a Q8_0 sidecar at temperature 0 (lossless).

Ports the NextN/MTP draft block for qwen4exp (one full layer, dense attention, own expert bank) and teaches the loader to accept mtp-only sidecar GGUFs: trunk tensors become optional when n_layer_nextn > 0 and blk.0.hc_attn_norm.weight is absent. Reference design: PR ggml-org#27739 (Qwen team); port template: LaurentZuijdwijk's vulkan/qwen4exp-rocmfpx branch. The unmasked nextn readback initially crashed: t_h_nextn carried only n_outputs rows after output masking while common/speculative reads ubatch.n_tokens rows. Output masking is now skipped while unmasked nextn embeddings are requested, following the deepseek4/deepseek32 pattern. Verified on gfx1151 (acceptance run 2026-08-28, 32K slot, Q8_0 sidecar, temperature 0): code decode 24.4 -> 34.3 t/s (+41%) at 79.6% draft acceptance; prose 22.2 t/s at 63.9%.

…aster), add the -ot host-pinning option from ggml-org#27739

…t is to beat llama.cpp at concurrency (localai-org#1995) `## Gates` item 5 said "Speed: nothing ... There is no vLLM denominator for this model, so when a speed axis does open, the spec must first say what the denominator is." A denominator does exist, the target for it lived only in a chat log, and `## Hardware` was still telling every parallel wave that the one artifact which fits does not exist. This says all three. **The denominator.** `llama-cpp` is already a registered oracle with a pin and `gateable = yes`, and its recorded scope is "GGUF k-quant speed and memory floors, quant-matched against the same weights". `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S is one published artifact both engines can run, which is exactly that condition. **The target, quoted rather than paraphrased,** because the wording is the requirement: "we should be faster than llama.cpp", and "especially at high concurrency". The gate is written to match. A c=1 number would neither confirm nor refute it, so a concurrency ladder is the headline; prefill and decode are reported separately because input length splits them; memory is an axis rather than a footnote; and llama.cpp runs in its production configuration, since this repository already has the `--enforce-eager` precedent for what a handicapped denominator does to a comparison. **The artifact exists.** `## Hardware` said `unsloth/Qwen3.8-Flash-Next-GGUF` was "README only, zero weight files -- does not exist". That was true when written and false a few hours later: three shards totalling 72,546,461,344 bytes = 67.56 GiB, fitting GB10 with roughly 52 GiB of headroom where every safetensors artifact does not fit at all. That row has been contradicting this spec's own later sections on `main` for two days, and the whole shape of `## Work breakdown` follows from the file existing, so the correction is called out in place rather than quietly applied. **Four levers say where the speed would come from,** from a source study of the two llama.cpp implementations of this architecture (ggml-org/llama.cpp#27742 open, ggml-org/llama.cpp#27739 closed by courtesy). Both are UNMERGED, and each lever is a reading of a pinned SHA rather than a measurement: continuous batching and paged KV against fixed parallel slots, the QSA gather consumer, n-gram table residency, and the hyper-connection write-back. **Merged forward over two days of waves, and three of its claims were stale on arrival.** A record that was true when written and is false now must not land as written, so each was corrected against the tree rather than carried. The two blockers this branch said stop us loading the file are both discharged by W6a (localai-org#2019), so the file opens today on the CPU arm and what remains owed is the CUDA half, where `DeviceQuantGatherSupported` is true for `kCPU` alone. W4 (localai-org#2030) built the gather rather than the mask, so the long-context lever this branch warned could be forfeited by accident was not. W3 (localai-org#2045) landed leaving `GatedResidualWriteBackInPlace` reachable and unfused. And the revision this branch recorded as still owed is now pinned in `## Owed`, leaving only a locally computed sha256 owed when W6 stages the file. The merge itself was verified rather than trusted, because AGENTS.md forbids accepting an automatic three-way merge of a keyed record: against `origin/main` this change removes exactly the eight lines it means to replace, and it leaves all six `## Mutation record` sections and all 53 `## Owed` bullets byte-for-byte intact. No number is claimed here. Nothing has been benchmarked, no arm runs yet, and the speed axis stays shut until G2 passes. The no-ceiling rule is restated in place, because the moment to write it down is before a first measurement disappoints rather than after. Records only: one file, `.agents/specs/qwen4-exp-flash-next.md`. Nothing under `src/`, `include/` or `tests/` changes, so this lands no code and owes no reachability proof. Closes localai-org#1990. Under localai-org#1978. Gates: `scripts/agent-preflight.sh --staged --fail-on-skip` rc=0, zero skips. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

…ut of a test helper, and the two settings a selection cannot see get a VALUE gate (localai-org#2211) (localai-org#2226) W5b-4 landed the claim that Qwen Sparse Attention's block score and top-k ARE `vt::DsaIndexerLogits` + `vt::DsaTopkSelect` with the per-head fold collapsed, and that collapse holds only under four settings: `weights` all ones, `n_head_scale == 1` rather than DeepSeek-V4's `n_head ** -0.5`, `softmax_scale == index_head_dim ** -0.5`, and `win_end == kv_len / compress_ratio` per query token. All four lived in the `RunIndexer` helper in `test_qwen4_exp_qsa_device.cpp`. Nothing under `src/` composed them, and the spec's `## Owed` said so in those words. `qwen4_exp_qsa_block.{h,cpp}` is that composition plus the block around it. `Qwen4ExpQsaIndex` states all four settings at one site, each beside the upstream line it mirrors and the mutation that reds it, and `RunQwen4ExpQsaBlock` is `Qwen4ExpTextAttention.forward` end to end — the q/k/v/o projections, the `(1 + w)`-polarity q/k norms, the RoPE, the indexer, the gather consumer, the sigmoid output gate and `o_proj`. It introduces no arithmetic; every step is a `vt::` primitive, which is what makes it a composition rather than a kernel. ## The value gate, and why a selection gate would not have been one Two of the four settings cannot be gated by a selection AT ALL. Top-k is invariant under a positive rescale of every score, so a wrong `n_head_scale` or `softmax_scale` moves nothing a selected set can show — which is why spec mutation M26 SURVIVED at the op layer in W5b-4, and why the debt was for a VALUE gate on the logits rather than for another selection comparison. `gen_qwen4_exp_qsa_block_goldens.py` captures the oracle's OWN pre-top-k `scores` tensor by intercepting `torch.Tensor.topk` inside the UNMODIFIED `Qwen4ExpTextQSAIndexer.forward` at the lane pin, transformers 5.16.0. It is a capture of the oracle's execution, not a transcription of its two scoring lines. Fed the oracle's own roped query and raw keys, the composed logits are BIT-IDENTICAL to it: max abs 0, over 12 and 60 logits at scales of 3.365 and 6.239. M1 (inherit `n_head ** -0.5`) and M2 (drop the softmax scale) each red that one case and nothing else, which is the repair M26 was owed. ## The oracle's identity The fresh review measured it rather than assuming it: the installed module was verified against the published 5.16.0 wheel's own RECORD hashes with 0 mismatches, and both this wave's goldens AND W4's already-landed goldens regenerate byte-identically from that module. In the tree the identity is executable rather than recorded — the generator refuses to emit under any `transformers.__version__` other than the lane pin, because a golden captured off an unpinned oracle is not reproducible. ## What the context length buys A QSA claim below the budget is not a claim: at or under `indexer_budget` every candidate is selected, so every read-count assertion is trivially true and a dense body passes it. The released-config case therefore runs at kv_len 3002, past the 2048 budget and deliberately not a multiple of the compress ratio so there is a ragged tail. 512 of 750 blocks are selected and the block reads `keys_visited` 16400 against a dense 24016. A NaN-poison probe at the BLOCK layer convicts a mask-shaped consumer that the golden comparison cannot see, since `exp(-inf - m)` is exactly +0 and a sparse mask over a dense cache agrees with a gather value for value — and under CUDA flash attention it costs the full dense prefix (`ggml-org/llama.cpp#27739`). ## The battery 24 mutations, every one proved applied by a sha256 that moved, every build return code read BEFORE its test result, the tree restored byte-for-byte and re-hashed, no survivors after one repair. M13 survived the first battery: writing the indexer key at cache row 0 instead of row `past_len` is invisible to a prefill, where the two spellings are the same expression, and the one decode case compared only the block OUTPUT at a bf16-sized bound, which one wrong pooled key out of five blocks stays inside. It is closed by an observable of WHERE the write landed rather than of what it was worth: the decode case now compares the side cache ROW FOR ROW against the oracle's own raw indexer keys, after a split prefill/decode. M3 and M4 first failed to BUILD under `-Werror`, which is a failure mode this campaign has now hit four times: ninja leaves the stale binary on disk and a stale binary prints green. Both were re-run with the one `(void)x;` or `[[maybe_unused]]` that silences the warning and changes nothing the mutation is about, and only the second reading is recorded. ## The fresh-review repair: a header that claimed a gate nobody had written The block takes one set of rope angles TWICE, in the two layouts its two ops were ported to read. The header said "the caller builds both from one table and the gate asserts they agree". Nothing asserted it, and the test's own `BuildRope` derives the packed cache FROM the full tables, so the two agree by construction — which is not an assertion. A layer loop that built them inconsistently would rope the query with one set of angles and the pooled indexer keys with another, silently, while the header told the next reader it was covered. Written rather than softened. `CheckRopeLayoutsAgree` refuses unequal heights, then compares a bounded sample of rows — row 1, the midpoint, the last row — value for value at one bf16 ulp. Row 0 is not a probe: cos is 1 and sin is 0 at every frequency there, so it agrees under every difference the probe exists to catch. A non-CPU-resident pair is refused BY NAME rather than skipped, because a check that silently does not run on a device arm is a mute switch. The full comparison is deliberately not paid: it would be O(P * rotary_dim) per QSA layer per step to re-check a constant. The red came first and it measured the silence. Two subcases were added before the check existed: one perturbs the full `cos` table at a row nothing else in the block reads, the other hands the two layouts different heights. On the pre-check head, build rc 0, both reported "did NOT throw at all" while the other 2829 of 2831 assertions passed. With the check, 8 of 8 cases and 2831 of 2831 assertions. A mutation deleting the production call site reds both subcases again, so they gate the CHECK and not the arithmetic. M1 and M13 were re-armed on the repaired head and red exactly as recorded, because a later commit can disarm an earlier commit's mutation proof. ## LANDS UNREACHED `qwen4_exp_qsa_block.{h,cpp}` is reached ONLY by its own test at this merge commit. Verified here rather than inherited: the only references to `RunQwen4ExpQsaBlock` and `Qwen4ExpQsaIndex` outside the pair itself are in `tests/vllm/models/test_qwen4_exp_qsa_block.cpp`, and `ForwardQwen4ExpForConditionalGeneration` still refuses `Qwen4ExpForConditionalGeneration` BY NAME because the layer loop is not written. The owning row is `MODEL-MM-QWEN4-EXP`, the wave issue is localai-org#2211, the forward is localai-org#2031, the campaign is localai-org#1978, and the spec's `## Owed` carries the unreached slice with the row and the issue that own the wiring. No reachability mutation is available at this layer, and that is stated in the mutation record rather than omitted: there is no production call site to delete. M20 — handing the consumer every VISIBLE block, a dense walk wearing a gather's clothes — is the strongest statement this slice can make, and it reds 3 of 8 cases. ## The trap the layer loop walks into `## Owed` now names the `hc_norm` polarity with the issue that owns it, localai-org#2218. `LoadNormBf16(..., unshift=true)` stores the RAW HuggingFace gamma, centred on 0, while `vt::Qwen4ExpGatedResidual` documents the opposite convention — "hc_norm_w is vLLM's parameterization, i.e. ALREADY `1 + w_hf` … This op never adds 1" — so a layer loop that wires them together applies a near-zero scale and the result reads as a corrupt checkpoint rather than as a wiring bug. The contradiction is visible AT THE LOAD SITE: the comment at `qwen4_exp_weights.cpp:258-263` argues FOR the fold, elementwise-corroborated on three published artifacts, immediately above the line that strips it. Not repaired here; it is its own wave, and the index row for localai-org#2218 names `MODEL-MM-QWEN4-EXP` as its owner. `## Owed` also gains two items that are new at the BLOCK level rather than inherited from the ops. `Qwen4ExpQsaIndex` builds `ones`, `win_start`, `win_end` and `kv_lens` on the host every call and reads `kv_lens` on the CPU behind a `VT_CHECK` — four small host-to-device copies per QSA layer per step on a device queue. And the pooled block keys are recomputed over the ENTIRE cache every step, O(kv) per layer per token. Upstream rebuilds the same metadata per call and recomputes the same pooled keys inside a per-query-token Python loop (`modeling_qwen4_exp.py:667-702` and `:679-686`), so both are faithful mirrors rather than divergences — and the nested Python loop is also the reminder that this oracle is a reference implementation and not a performance model. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

…t is to beat llama.cpp at concurrency (localai-org#1995) `## Gates` item 5 said "Speed: nothing ... There is no vLLM denominator for this model, so when a speed axis does open, the spec must first say what the denominator is." A denominator does exist, the target for it lived only in a chat log, and `## Hardware` was still telling every parallel wave that the one artifact which fits does not exist. This says all three. **The denominator.** `llama-cpp` is already a registered oracle with a pin and `gateable = yes`, and its recorded scope is "GGUF k-quant speed and memory floors, quant-matched against the same weights". `unsloth/Qwen3.8-Flash-Next-GGUF` UD-IQ1_S is one published artifact both engines can run, which is exactly that condition. **The target, quoted rather than paraphrased,** because the wording is the requirement: "we should be faster than llama.cpp", and "especially at high concurrency". The gate is written to match. A c=1 number would neither confirm nor refute it, so a concurrency ladder is the headline; prefill and decode are reported separately because input length splits them; memory is an axis rather than a footnote; and llama.cpp runs in its production configuration, since this repository already has the `--enforce-eager` precedent for what a handicapped denominator does to a comparison. **The artifact exists.** `## Hardware` said `unsloth/Qwen3.8-Flash-Next-GGUF` was "README only, zero weight files -- does not exist". That was true when written and false a few hours later: three shards totalling 72,546,461,344 bytes = 67.56 GiB, fitting GB10 with roughly 52 GiB of headroom where every safetensors artifact does not fit at all. That row has been contradicting this spec's own later sections on `main` for two days, and the whole shape of `## Work breakdown` follows from the file existing, so the correction is called out in place rather than quietly applied. **Four levers say where the speed would come from,** from a source study of the two llama.cpp implementations of this architecture (ggml-org/llama.cpp#27742 open, ggml-org/llama.cpp#27739 closed by courtesy). Both are UNMERGED, and each lever is a reading of a pinned SHA rather than a measurement: continuous batching and paged KV against fixed parallel slots, the QSA gather consumer, n-gram table residency, and the hyper-connection write-back. **Merged forward over two days of waves, and three of its claims were stale on arrival.** A record that was true when written and is false now must not land as written, so each was corrected against the tree rather than carried. The two blockers this branch said stop us loading the file are both discharged by W6a (localai-org#2019), so the file opens today on the CPU arm and what remains owed is the CUDA half, where `DeviceQuantGatherSupported` is true for `kCPU` alone. W4 (localai-org#2030) built the gather rather than the mask, so the long-context lever this branch warned could be forfeited by accident was not. W3 (localai-org#2045) landed leaving `GatedResidualWriteBackInPlace` reachable and unfused. And the revision this branch recorded as still owed is now pinned in `## Owed`, leaving only a locally computed sha256 owed when W6 stages the file. The merge itself was verified rather than trusted, because AGENTS.md forbids accepting an automatic three-way merge of a keyed record: against `origin/main` this change removes exactly the eight lines it means to replace, and it leaves all six `## Mutation record` sections and all 53 `## Owed` bullets byte-for-byte intact. No number is claimed here. Nothing has been benchmarked, no arm runs yet, and the speed axis stays shut until G2 passes. The no-ceiling rule is restated in place, because the moment to write it down is before a first measurement disappoints rather than after. Records only: one file, `.agents/specs/qwen4-exp-flash-next.md`. Nothing under `src/`, `include/` or `tests/` changes, so this lands no code and owes no reachability proof. Closes localai-org#1990. Under localai-org#1978. Gates: `scripts/agent-preflight.sh --staged --fail-on-skip` rc=0, zero skips. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [Claude Code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

…ut of a test helper, and the two settings a selection cannot see get a VALUE gate (localai-org#2211) (localai-org#2226) W5b-4 landed the claim that Qwen Sparse Attention's block score and top-k ARE `vt::DsaIndexerLogits` + `vt::DsaTopkSelect` with the per-head fold collapsed, and that collapse holds only under four settings: `weights` all ones, `n_head_scale == 1` rather than DeepSeek-V4's `n_head ** -0.5`, `softmax_scale == index_head_dim ** -0.5`, and `win_end == kv_len / compress_ratio` per query token. All four lived in the `RunIndexer` helper in `test_qwen4_exp_qsa_device.cpp`. Nothing under `src/` composed them, and the spec's `## Owed` said so in those words. `qwen4_exp_qsa_block.{h,cpp}` is that composition plus the block around it. `Qwen4ExpQsaIndex` states all four settings at one site, each beside the upstream line it mirrors and the mutation that reds it, and `RunQwen4ExpQsaBlock` is `Qwen4ExpTextAttention.forward` end to end — the q/k/v/o projections, the `(1 + w)`-polarity q/k norms, the RoPE, the indexer, the gather consumer, the sigmoid output gate and `o_proj`. It introduces no arithmetic; every step is a `vt::` primitive, which is what makes it a composition rather than a kernel. ## The value gate, and why a selection gate would not have been one Two of the four settings cannot be gated by a selection AT ALL. Top-k is invariant under a positive rescale of every score, so a wrong `n_head_scale` or `softmax_scale` moves nothing a selected set can show — which is why spec mutation M26 SURVIVED at the op layer in W5b-4, and why the debt was for a VALUE gate on the logits rather than for another selection comparison. `gen_qwen4_exp_qsa_block_goldens.py` captures the oracle's OWN pre-top-k `scores` tensor by intercepting `torch.Tensor.topk` inside the UNMODIFIED `Qwen4ExpTextQSAIndexer.forward` at the lane pin, transformers 5.16.0. It is a capture of the oracle's execution, not a transcription of its two scoring lines. Fed the oracle's own roped query and raw keys, the composed logits are BIT-IDENTICAL to it: max abs 0, over 12 and 60 logits at scales of 3.365 and 6.239. M1 (inherit `n_head ** -0.5`) and M2 (drop the softmax scale) each red that one case and nothing else, which is the repair M26 was owed. ## The oracle's identity The fresh review measured it rather than assuming it: the installed module was verified against the published 5.16.0 wheel's own RECORD hashes with 0 mismatches, and both this wave's goldens AND W4's already-landed goldens regenerate byte-identically from that module. In the tree the identity is executable rather than recorded — the generator refuses to emit under any `transformers.__version__` other than the lane pin, because a golden captured off an unpinned oracle is not reproducible. ## What the context length buys A QSA claim below the budget is not a claim: at or under `indexer_budget` every candidate is selected, so every read-count assertion is trivially true and a dense body passes it. The released-config case therefore runs at kv_len 3002, past the 2048 budget and deliberately not a multiple of the compress ratio so there is a ragged tail. 512 of 750 blocks are selected and the block reads `keys_visited` 16400 against a dense 24016. A NaN-poison probe at the BLOCK layer convicts a mask-shaped consumer that the golden comparison cannot see, since `exp(-inf - m)` is exactly +0 and a sparse mask over a dense cache agrees with a gather value for value — and under CUDA flash attention it costs the full dense prefix (`ggml-org/llama.cpp#27739`). ## The battery 24 mutations, every one proved applied by a sha256 that moved, every build return code read BEFORE its test result, the tree restored byte-for-byte and re-hashed, no survivors after one repair. M13 survived the first battery: writing the indexer key at cache row 0 instead of row `past_len` is invisible to a prefill, where the two spellings are the same expression, and the one decode case compared only the block OUTPUT at a bf16-sized bound, which one wrong pooled key out of five blocks stays inside. It is closed by an observable of WHERE the write landed rather than of what it was worth: the decode case now compares the side cache ROW FOR ROW against the oracle's own raw indexer keys, after a split prefill/decode. M3 and M4 first failed to BUILD under `-Werror`, which is a failure mode this campaign has now hit four times: ninja leaves the stale binary on disk and a stale binary prints green. Both were re-run with the one `(void)x;` or `[[maybe_unused]]` that silences the warning and changes nothing the mutation is about, and only the second reading is recorded. ## The fresh-review repair: a header that claimed a gate nobody had written The block takes one set of rope angles TWICE, in the two layouts its two ops were ported to read. The header said "the caller builds both from one table and the gate asserts they agree". Nothing asserted it, and the test's own `BuildRope` derives the packed cache FROM the full tables, so the two agree by construction — which is not an assertion. A layer loop that built them inconsistently would rope the query with one set of angles and the pooled indexer keys with another, silently, while the header told the next reader it was covered. Written rather than softened. `CheckRopeLayoutsAgree` refuses unequal heights, then compares a bounded sample of rows — row 1, the midpoint, the last row — value for value at one bf16 ulp. Row 0 is not a probe: cos is 1 and sin is 0 at every frequency there, so it agrees under every difference the probe exists to catch. A non-CPU-resident pair is refused BY NAME rather than skipped, because a check that silently does not run on a device arm is a mute switch. The full comparison is deliberately not paid: it would be O(P * rotary_dim) per QSA layer per step to re-check a constant. The red came first and it measured the silence. Two subcases were added before the check existed: one perturbs the full `cos` table at a row nothing else in the block reads, the other hands the two layouts different heights. On the pre-check head, build rc 0, both reported "did NOT throw at all" while the other 2829 of 2831 assertions passed. With the check, 8 of 8 cases and 2831 of 2831 assertions. A mutation deleting the production call site reds both subcases again, so they gate the CHECK and not the arithmetic. M1 and M13 were re-armed on the repaired head and red exactly as recorded, because a later commit can disarm an earlier commit's mutation proof. ## LANDS UNREACHED `qwen4_exp_qsa_block.{h,cpp}` is reached ONLY by its own test at this merge commit. Verified here rather than inherited: the only references to `RunQwen4ExpQsaBlock` and `Qwen4ExpQsaIndex` outside the pair itself are in `tests/vllm/models/test_qwen4_exp_qsa_block.cpp`, and `ForwardQwen4ExpForConditionalGeneration` still refuses `Qwen4ExpForConditionalGeneration` BY NAME because the layer loop is not written. The owning row is `MODEL-MM-QWEN4-EXP`, the wave issue is localai-org#2211, the forward is localai-org#2031, the campaign is localai-org#1978, and the spec's `## Owed` carries the unreached slice with the row and the issue that own the wiring. No reachability mutation is available at this layer, and that is stated in the mutation record rather than omitted: there is no production call site to delete. M20 — handing the consumer every VISIBLE block, a dense walk wearing a gather's clothes — is the strongest statement this slice can make, and it reds 3 of 8 cases. ## The trap the layer loop walks into `## Owed` now names the `hc_norm` polarity with the issue that owns it, localai-org#2218. `LoadNormBf16(..., unshift=true)` stores the RAW HuggingFace gamma, centred on 0, while `vt::Qwen4ExpGatedResidual` documents the opposite convention — "hc_norm_w is vLLM's parameterization, i.e. ALREADY `1 + w_hf` … This op never adds 1" — so a layer loop that wires them together applies a near-zero scale and the result reads as a corrupt checkpoint rather than as a wiring bug. The contradiction is visible AT THE LOAD SITE: the comment at `qwen4_exp_weights.cpp:258-263` argues FOR the fold, elementwise-corroborated on three published artifacts, immediately above the line that strips it. Not repaired here; it is its own wave, and the index row for localai-org#2218 names `MODEL-MM-QWEN4-EXP` as its owner. `## Owed` also gains two items that are new at the BLOCK level rather than inherited from the ops. `Qwen4ExpQsaIndex` builds `ones`, `win_start`, `win_end` and `kv_lens` on the host every call and reads `kv_lens` on the CPU behind a `VT_CHECK` — four small host-to-device copies per QSA layer per step on a device queue. And the pooled block keys are recomputed over the ENTIRE cache every step, O(kv) per layer per token. Upstream rebuilds the same metadata per call and recomputes the same pooled keys inside a per-query-token Python loop (`modeling_qwen4_exp.py:667-702` and `:679-686`), so both are faithful mirrors rather than divergences — and the nested Python loop is also the reminder that this oracle is a reference implementation and not a performance model. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

Ports the NextN/MTP draft block for qwen4exp (one full layer, dense attention, own expert bank) and teaches the loader to accept mtp-only sidecar GGUFs: trunk tensors become optional when n_layer_nextn > 0 and blk.0.hc_attn_norm.weight is absent. Reference design: PR ggml-org#27739 (Qwen team); port template: LaurentZuijdwijk's vulkan/qwen4exp-rocmfpx branch. The unmasked nextn readback initially crashed: t_h_nextn carried only n_outputs rows after output masking while common/speculative reads ubatch.n_tokens rows. Output masking is now skipped while unmasked nextn embeddings are requested, following the deepseek4/deepseek32 pattern. Verified on gfx1151 (acceptance run 2026-08-28, 32K slot, Q8_0 sidecar, temperature 0): code decode 24.4 -> 34.3 t/s (+41%) at 79.6% draft acceptance; prose 22.2 t/s at 63.9%.

…aster), add the -ot host-pinning option from ggml-org#27739

Upstream's qwen4exp (ggml-org#27742) shipped without MTP -- the author had it as WIP, and the competing ggml-org#27739, which did implement it, was closed. No follow-up PR is open and no MTP GGUF is published anywhere, so both the converter path and the graph are added here. The reference is ggml-org#27739, adapted rather than cherry-picked: it targets a different implementation of the same architecture. Converter: drop upstream's `no_mtp` opt-out (the shared _QwenMtpMixin already sits in qwen4exp's MRO via _LinearAttentionVReorderBase -> Qwen3NextModel, so enabling export is a deletion). Fuse the reference's two MTP input projections into the single eh_proj the tensor map knows -- A*e + B*h == [A|B]*concat(e,h). Keep the MTP block's own hyper-connection mixer by renaming it into the model-level slot before the mixin drops it as a non-MTP tensor. An --mtp export emits no PLE keys, and the compress-ratio array is padded to block_count: the MTP block runs its attention dense, so it takes ratio 0. Graph: one MTP block at index n_layer, fed the trunk's hyper-connection streams rather than the collapsed hidden state. Two things this needs that the trunk did not previously provide: - The trunk now exports res->t_h_nextn. The draft reads it via the driver's masked=false contract, so the last layer's inp_out_ids reduction is deferred until after the export, and moves as a unit -- build_hc_combine sizes itself from the residual, so reducing cur/inject while res_hc stays full width fails a reshape. The reduction is chained off the exported tensor because an exported view that nothing consumes is a dead node, and llama-context then asserts on the missing backend when reading h_nextn back. - An MTP-only draft declares n_embd_inp_impl = n_embd_out(). DeepSeek-V4 gets this for free by giving its draft a separate arch (deepseek4mtp, whose embedding_length is already the wide value); qwen4exp uses one arch for both files, so the draft has to widen its own input row or common/speculative.cpp rejects the pair. QWEN4EXP joins mtp_on_hybrid_qwen so the MTP context gets a plain attention KV cache holding only the nextn layer. Without it the draft allocated a cache for the trunk layers it does not carry and none for the block it does. build_layer_attn now tolerates a null hybrid context: the MTP block runs dense. Measured on GB10, UD-IQ4_XS target + Q8_0 draft, c32768/ub1024, 3 prompts x 400 tokens: no MTP 18.45 t/s; n_max=1 25.17 t/s at acceptance 0.9179 (+36.4%); n_max=2 20.80 t/s at 0.5807. n=1 is the operating point -- position-1 acceptance backs out to ~0.24 while widening the verify batch. Acceptance is sensitive to output length: the same build reads 0.4583 on a 64-token summary. Not gated: the DeepSeek-V4 perplexity anchor has not been re-run against the trunk change. defer_out_ids is false whenever embeddings_nextn is off, so the non-MTP path should be untouched, but that is an argument and not a measurement.

Ports the NextN/MTP draft block for qwen4exp (one full layer, dense attention, own expert bank) and teaches the loader to accept mtp-only sidecar GGUFs: trunk tensors become optional when n_layer_nextn > 0 and blk.0.hc_attn_norm.weight is absent. Reference design: PR ggml-org#27739 (Qwen team); port template: LaurentZuijdwijk's vulkan/qwen4exp-rocmfpx branch. The unmasked nextn readback initially crashed: t_h_nextn carried only n_outputs rows after output masking while common/speculative reads ubatch.n_tokens rows. Output masking is now skipped while unmasked nextn embeddings are requested, following the deepseek4/deepseek32 pattern. Verified on gfx1151 (acceptance run 2026-08-28, 32K slot, Q8_0 sidecar, temperature 0): code decode 24.4 -> 34.3 t/s (+41%) at 79.6% draft acceptance; prose 22.2 t/s at 63.9%.

…aster), add the -ot host-pinning option from ggml-org#27739

Upstream's qwen4exp (ggml-org#27742) shipped without MTP -- the author had it as WIP, and the competing ggml-org#27739, which did implement it, was closed. No follow-up PR is open and no MTP GGUF is published anywhere, so both the converter path and the graph are added here. The reference is ggml-org#27739, adapted rather than cherry-picked: it targets a different implementation of the same architecture. Converter: drop upstream's `no_mtp` opt-out (the shared _QwenMtpMixin already sits in qwen4exp's MRO via _LinearAttentionVReorderBase -> Qwen3NextModel, so enabling export is a deletion). Fuse the reference's two MTP input projections into the single eh_proj the tensor map knows -- A*e + B*h == [A|B]*concat(e,h). Keep the MTP block's own hyper-connection mixer by renaming it into the model-level slot before the mixin drops it as a non-MTP tensor. An --mtp export emits no PLE keys, and the compress-ratio array is padded to block_count: the MTP block runs its attention dense, so it takes ratio 0. Graph: one MTP block at index n_layer, fed the trunk's hyper-connection streams rather than the collapsed hidden state. Two things this needs that the trunk did not previously provide: - The trunk now exports res->t_h_nextn. The draft reads it via the driver's masked=false contract, so the last layer's inp_out_ids reduction is deferred until after the export, and moves as a unit -- build_hc_combine sizes itself from the residual, so reducing cur/inject while res_hc stays full width fails a reshape. The reduction is chained off the exported tensor because an exported view that nothing consumes is a dead node, and llama-context then asserts on the missing backend when reading h_nextn back. - An MTP-only draft declares n_embd_inp_impl = n_embd_out(). DeepSeek-V4 gets this for free by giving its draft a separate arch (deepseek4mtp, whose embedding_length is already the wide value); qwen4exp uses one arch for both files, so the draft has to widen its own input row or common/speculative.cpp rejects the pair. QWEN4EXP joins mtp_on_hybrid_qwen so the MTP context gets a plain attention KV cache holding only the nextn layer. Without it the draft allocated a cache for the trunk layers it does not carry and none for the block it does. build_layer_attn now tolerates a null hybrid context: the MTP block runs dense. Measured on GB10, UD-IQ4_XS target + Q8_0 draft, c32768/ub1024, 3 prompts x 400 tokens: no MTP 18.45 t/s; n_max=1 25.17 t/s at acceptance 0.9179 (+36.4%); n_max=2 20.80 t/s at 0.5807. n=1 is the operating point -- position-1 acceptance backs out to ~0.24 while widening the verify batch. Acceptance is sensitive to output length: the same build reads 0.4583 on a 64-token summary. Not gated: the DeepSeek-V4 perplexity anchor has not been re-run against the trunk change. defer_out_ids is false whenever embeddings_nextn is off, so the non-MTP path should be untouched, but that is an argument and not a measurement.

Ports the NextN/MTP draft block for qwen4exp (one full layer, dense attention, own expert bank) and teaches the loader to accept mtp-only sidecar GGUFs: trunk tensors become optional when n_layer_nextn > 0 and blk.0.hc_attn_norm.weight is absent. Reference design: PR ggml-org#27739 (Qwen team); port template: LaurentZuijdwijk's vulkan/qwen4exp-rocmfpx branch. The unmasked nextn readback initially crashed: t_h_nextn carried only n_outputs rows after output masking while common/speculative reads ubatch.n_tokens rows. Output masking is now skipped while unmasked nextn embeddings are requested, following the deepseek4/deepseek32 pattern. Verified on gfx1151 (acceptance run 2026-08-28, 32K slot, Q8_0 sidecar, temperature 0): code decode 24.4 -> 34.3 t/s (+41%) at 79.6% draft acceptance; prose 22.2 t/s at 63.9%.

…aster), add the -ot host-pinning option from ggml-org#27739

Ports the NextN/MTP draft block for qwen4exp (one full layer, dense attention, own expert bank) and teaches the loader to accept mtp-only sidecar GGUFs: trunk tensors become optional when n_layer_nextn > 0 and blk.0.hc_attn_norm.weight is absent. Reference design: PR ggml-org#27739 (Qwen team); port template: LaurentZuijdwijk's vulkan/qwen4exp-rocmfpx branch. The unmasked nextn readback initially crashed: t_h_nextn carried only n_outputs rows after output masking while common/speculative reads ubatch.n_tokens rows. Output masking is now skipped while unmasked nextn embeddings are requested, following the deepseek4/deepseek32 pattern. Verified on gfx1151 (acceptance run 2026-08-28, 32K slot, Q8_0 sidecar, temperature 0): code decode 24.4 -> 34.3 t/s (+41%) at 79.6% draft acceptance; prose 22.2 t/s at 63.9%.

…aster), add the -ot host-pinning option from ggml-org#27739

model : add Qwen4-Exp

dfa0c0f

JJJYmmm requested review from CISC and ggerganov as code owners August 26, 2026 13:28

github-actions Bot added model Model specific conversion labels Aug 26, 2026

This comment was marked as resolved.

Sign in to view

ggml-gh-bot Bot added the draft PR will be changed to draft by github-actions bot label Aug 26, 2026

github-actions Bot marked this pull request as draft August 26, 2026 13:33

github-actions Bot removed the draft PR will be changed to draft by github-actions bot label Aug 26, 2026

ggerganov marked this pull request as ready for review August 26, 2026 13:33

ggerganov self-assigned this Aug 26, 2026

JJJYmmm closed this Aug 26, 2026

JJJYmmm mentioned this pull request Aug 26, 2026

model: add Qwen3.8-Flash-Next (qwen4exp) #27742

Merged

CISC reviewed Aug 26, 2026

View reviewed changes

rmonsurate mentioned this pull request Aug 27, 2026

qwen4exp : add NextN/MTP draft head (--spec-type draft-mtp) for Qwen3.8-Flash-Next #27836

Draft

1 task done

routhjim mentioned this pull request Aug 28, 2026

model : add MTP (nextn) speculative head for Qwen3.8-Flash-Next (qwen4exp) #27842

Closed

Aristo94 added a commit to Aristo94/EngramHalo.cpp that referenced this pull request Aug 28, 2026

docs: precise citation for Unsloth's mmap prefetch (95da4ba, not in m…

9ea6b77

…aster), add the -ot host-pinning option from ggml-org#27739

routhjim mentioned this pull request Aug 31, 2026

model : add MTP (nextn) speculative head for Qwen3.8-Flash-Next (qwen4exp) routhjim/llama.cpp-lab#1

Merged

Aristo94 added a commit to Aristo94/EngramHalo.cpp that referenced this pull request Aug 31, 2026

docs: precise citation for Unsloth's mmap prefetch (95da4ba, not in m…

37556cc

…aster), add the -ot host-pinning option from ggml-org#27739

Aristo94 added a commit to Aristo94/EngramHalo.cpp that referenced this pull request Sep 3, 2026

docs: precise citation for Unsloth's mmap prefetch (95da4ba, not in m…

556d64f

…aster), add the -ot host-pinning option from ggml-org#27739

Aristo94 added a commit to Aristo94/EngramHalo.cpp that referenced this pull request Sep 28, 2026

docs: precise citation for Unsloth's mmap prefetch (95da4ba, not in m…

0d19fdb

…aster), add the -ot host-pinning option from ggml-org#27739

Aristo94 added a commit to Aristo94/EngramHalo.cpp that referenced this pull request Sep 30, 2026

docs: precise citation for Unsloth's mmap prefetch (95da4ba, not in m…

130738c

…aster), add the -ot host-pinning option from ggml-org#27739

Navigation Menu

Repository navigation

model : add Qwen4-Exp - #27739

model : add Qwen4-Exp#27739
JJJYmmm wants to merge 1 commit into
ggml-org:masterfrom
JJJYmmm:add_qwen4exp

JJJYmmm commented Aug 26, 2026 •

edited

Loading

Uh oh!

This comment was marked as resolved.

JJJYmmm commented Aug 26, 2026

Uh oh!

danielhanchen commented Aug 26, 2026 •

edited

Loading

Uh oh!

JJJYmmm commented Aug 26, 2026

Uh oh!

JJJYmmm commented Aug 26, 2026

Uh oh!

danielhanchen commented Aug 26, 2026

Uh oh!

CISC Aug 26, 2026

Uh oh!

CISC Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	for (int il = n_layer; il < n_layer + (int) hparams.n_layer_nextn; ++il) {
	for (int il = n_layer; il < n_layer_all; ++il) {

Conversation

JJJYmmm commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Not supported

Uh oh!

This comment was marked as resolved.

JJJYmmm commented Aug 26, 2026

Uh oh!

danielhanchen commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JJJYmmm commented Aug 26, 2026

Uh oh!

JJJYmmm commented Aug 26, 2026

Uh oh!

danielhanchen commented Aug 26, 2026

Uh oh!

CISC Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

CISC Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JJJYmmm commented Aug 26, 2026 •

edited

Loading

danielhanchen commented Aug 26, 2026 •

edited

Loading