model: DSV4.1 flash by am17an · Pull Request #31086 · ggml-org/llama.cpp

am17an · 2026-10-24T10:09:26Z

Overview

Additional information

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES

…_kv_cache::seq_pos_tok_le

…host gather

…ule per Engram layer

pwilkin · 2026-10-24T12:18:56Z

/bot review

ggml-gh-bot · 2026-10-24T12:18:58Z

Automated code review

Review: model: DSV4.1 flash (PR #31086)

Scope: new model architecture (DEEPSEEK41) plus conversion, chat template handling, DSpark draft (DFLASH), mtmd vision, DSV4 KV-cache extensions, and small ggml numeric tweaks. Overall the change follows the existing DSV4/qwen4exp patterns closely, hparams from GGUF are validated well (bounds on engram bucket offsets/primes are checked before any row gather, so crafted metadata cannot index out of the table), and test coverage is meaningful (test-chat V4.1 template cases, test-llama-archs fixture, im2col 3x3 stride-3 op test). Findings below.

Blocking / process

(point 1) The PR description is empty: no overview, no linked issue/discussion, and the AI-usage disclosure section of the template is unfilled. Per CONTRIBUTING.md, features are supposed to start from an issue, and the disclosure requirement is explicit. Please fill these in yourself before maintainers review; an empty description risks summary rejection regardless of code quality.

Correctness / robustness

(point 2) tools/mtmd/mtmd-image.cpp:1475 - GGML_ASSERT(n_tokens(best_height, best_width) <= max_n_token) at the end of plan_grid. The plan is derived from user-supplied image dimensions, and the solve_resize_ratio port uses floating point (sqrt can round up by an ulp at exact boundaries), so a crafted image size can turn this into a process abort on a server. Clamp instead of assert (e.g. shrink by one cell on failure), or compute the bound in integer math.

(point 3) src/llama-kv-cache-dsv4.cpp:440 - the mirror refill in dsv4_state_read_k_cache uses GGML_ASSERT for the type/width match in a state-restore path where every other failure throws std::runtime_error. Per project convention (loaders/state IO throw, callers catch), prefer a throw here.

(point 4) LLAMA_MAX_PLE_NGRAM == 8 / LLAMA_MAX_PLE_HEADS == 64 now bound the totals across all Engram modules (mult.size() > LLAMA_MAX_PLE_NGRAM, primes.size() > LLAMA_MAX_PLE_HEADS in deepseek41.cpp load), but the underlying hparams are fixed std::arrays of those sizes. Please verify the released DeepSeek-V4.1-Flash-1017 config actually fits (n_engram_layers * max_ngram_size <= 8, and n_engram_layers * (ngram-1) * n_heads <= 64); if a later variant exceeds it, the converted GGUF fails to load with "engram metadata is inconsistent". State the constraint in the hparams comment.

(point 5) src/llama-kv-cache.cpp:332 - DEEPSEEK41 is not in the "always create Hadamard rotation for lightning indexers" list (DEEPSEEK32/DEEPSEEK4/GLM_DSA/DOTS3NOTE are). If that is intentional because n_embd_head_k_full == indexer_head_size is false for V4.1, fine; if the sizes are equal, quantized-KV loses the incoherence rotation V4 gets (quality regression for -ctk q8_0). Please confirm and, if deliberate, add a one-line comment.

(point 6) src/llama-model-saver.cpp:337-342 - the saver writes the other HYPER_CONNECTION keys but not the new LLM_KV_HYPER_CONNECTION_SINGLE_PASS (hparams.dsv4_hc_single_pass), which the DFLASH loader reads. A V4.1 DSpark draft round-tripped through the model saver would silently dispatch to graph_dsv4, which then requires the hc_head tensors that do not exist. One-line fix next to the other hyper-connection keys. (Note: the saver already omits several DFLASH-specific keys, so DSpark saving may be incomplete as-is - if so, worth fixing as a group, but the new key should not add to the gap.)

(point 7) src/llama-context.cpp:1342 - the set_embeddings change now sets sched_need_reserve = true on every toggle for every model, not just pooled contexts. The comment explains a real gallocr crash, but this is a global behavior change bundled into a model PR; it deserves a line in the PR description, and ideally a check that the frequent reserve does not regress toggling-heavy workloads.

(point 8) The log1pf(expf(x)) softplus change (ggml/src/ggml-cpu/unary-ops.cpp, ggml/src/ggml-cuda/unary.cu, ggml/src/ggml-cuda/topk-moe.cu) alters numerics of an existing public ggml op for all models. It is strictly more accurate and backends stay consistent, but ggml-core changes carry a higher bar and are usually split out - please move it to its own tiny PR (or call it out explicitly here and note the expected two-maintainer signoff for ggml/).

Review speed / conventions

(point 9) The V4.1 detection literal "｜DSML｜ calls>" is duplicated in three places (common/chat.cpp template detect, common/chat.cpp apply, common/parsers/deepseek.cpp is_v41). Hoist it into one shared constant so the three checks cannot drift.

(point 10) common/chat.h - common_chat_msg::operator== does not compare the new extra_json field; message-equality-based tests will not catch extras regressions (the render tests do, but the asymmetry is worth a look).

(point 11) Two full attention code paths now exist, selected by LLAMA_DSV41_SPARSE (default ON = optimized gather; =0 = dense "reference" path), plus LLAMA_DSV41_KV_MIRROR for layer splits. The dense path is only reachable via env var, so CI will only ever run the sparse one. Please make sure at least one test/run covers LLAMA_DSV41_SPARSE=0 and the quantized-KV path (-ctk/-ctv q8_0) - the gather/dense branches contain several type-dependent special cases (k_rot handling, quantized REPEAT workaround via concat doubling, strided-view dequant) that silently bit-rot otherwise.

(point 12) common/parsers/deepseek.cpp - the reasoning_effort comment says "medium -> default" but the code writes a JSON null (suppressing the effort tag entirely). The test asserts "" so the behavior is presumably right, but the comment should say "unset / no effort tag" to avoid confusion.

(point 13) src/llama-kv-cache-dsv4.cpp:1424 - for V4.1 the lightning-indexer cache and compressor state are still created (and logged) with zero layers, and lid plans/inputs are still built per batch and never used by the V4.1 graph. Consider skipping them entirely when is_v41, or note why they must exist.

(point 14) Converter nits: DeepseekV41VisionModel.set_gguf_parameters uses bare assert for config validation while the rest of the new file raises ValueError - pick one style; _StreamedBytes.byteswap() raising NotImplementedError mid-write on a big-endian host leaves a corrupt partial file - failing loudly before any write would be friendlier.

What looks good

Engram GGUF metadata is bounded thoroughly before use (bucket range, per-layer ordering, table row count vs offsets), and the host-side hash cannot produce out-of-range row ids even with hostile token_map values.
The im2col kernel-view stride fix in tools/mtmd/models/deepseek4v.cpp is correct: im2col only reads src1, so the strided view is output-equivalent and unlocks the CUDA path.
The V4 state-IO format is preserved exactly for V4 (mirrors are excluded from serialization and refilled on read), so old state files stay compatible.
dsv41_shared_history_ok correctly rejects (rather than silently mis-hashing) batches whose shared tokens have divergent n-gram histories, with a clear error.

This review was generated automatically by pi coding agent using zai-org/GLM-5.3. It may contain mistakes. Maintainers make the final call.

am17an · 2026-10-24T13:47:40Z

okay this should be a good shape now. @ngxson can you review the mtmd code? otherwise I think it looks ok - I'm yet to dive deep into the model optimization but will do in the future.

pwilkin

Had to add a small patch to adapt the GSQ GGUFs for this one, but I ran it on my R9700+5060+128 GB RAM and it works, even if prefill is somewhat disappointing right now (decode's quite decent though):

14.11.934.556 I slot print_timing: id  0 | task 0 | prompt eval time =  265590.08 ms /  8318 tokens (   31.93 ms per token,    31.32 tokens per second)
14.11.934.560 I slot print_timing: id  0 | task 0 |        eval time =  129614.12 ms /   850 tokens (  152.67 ms per token,     6.55 tokens per second)
14.11.934.560 I slot print_timing: id  0 | task 0 |       total time =  395204.20 ms /  9168 tokens
14.11.936.252 I slot print_timing: id  0 | task 0 |    graphs reused =        845
14.11.936.253 I slot print_timing: id  0 | task 0 |   graph reallocs =         20    /     0        (same size) / (changed size)

ngxson

mtmd part looks fine, optionally may worth adding some test cases to test-mtmd-impl

Upstream ggml-org#31086 changed the CPU and CUDA SOFTPLUS op to log1pf(expf(x)). The fused alpha gate kernels (MMVF and MMVQ) still used logf(1 + expf(x)), so decode/verify (fused) and prefill (unfused) computed different softplus values after the 2026-10-28 sync: ubatch 4 KLD with GGML_HIP_GDN_GATES=0 vs default was 0.003962 (top-1 97.6%). With the same formula the two paths match again (KLD 0, max 0.000052 = run-to-run floor).

The small kernels between the qkv/z and ssm_out mat-vecs of a GDN layer (14 per layer in decode) are now 4: - get_rows + concat + cpy + ssm_conv + silu -> ssm_conv_state_f32 (single state row, d_conv 4, no bias) - the recurrent state is read in the GDN kernel through the ids tensor - the q/k l2 norms, softplus*A gate and sigmoid(beta) are folded into the GDN kernel (S_v 128, warp 32) - rms_norm * weight * silu(z) -> rms_norm_mul_gated_f32 qwen35, qwen35moe and qwen3next order their graph nodes to match. Other shapes keep the old kernels. tests: GDN decode fusion, RMS_NORM_MUL_GATED. Per GDN layer 27B tensor 58.7 -> 21.3 us; kernels per token 1676 -> 1210. tg128: 27B Q8_0 tensor +8.6%, layer +4.8%, 27B IQ4_XS +3.8%, 4B +7.6%, 35B-A3B tensor +11.5%, Coder-Next 80B tensor +6.5%; prefill flat. test-backend-ops 19011/19011, PPL and greedy output identical. p100-series port (v5.34): fused softplus uses upstream ggml-org#31086's log1pf like op_softplus; the Ampere MMA GDN path is skipped when the inputs are fused; Kane's Pascal chunked-prefill hunk dropped (not carried). Cherry-picked-from: KaneQuah@074fa77 Assisted-by: Claude Opus 5.5

* model : add DeepSeek-V4.1-Flash (deepseek41) * deepseek41 : remove MXFP8, convert the Engram table to BF16 * deepseek41 : compute the compressed-KV read layers once in the cache * deepseek41 : use get_cells() for the shared-history check, drop llama_kv_cache::seq_pos_tok_le * deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the host gather * deepseek41 : keep the Engram hash constants in hparams.ple_*, one module per Engram layer * deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle * deepseek41 : keep il last in build_hc_mixes * deepseek41 : sort the top-k rows before the sparse gather * tests : add deepseek41 to the Metal fusion baseline --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com>

Fills in what upstream ggml-org#31086 builds on: the Engram n-gram history lookup in the KV cells, row prefetch for the memory-mapped Engram table, the hyper-connection stream mean, the vision router bias tensor, and the current build_attn_mha / build_moe_ffn signatures. The per-op precision requests map onto the mul_mat accumulator setting this ggml has. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The small kernels between the qkv/z and ssm_out mat-vecs of a GDN layer (14 per layer in decode) are now 4: - get_rows + concat + cpy + ssm_conv + silu -> ssm_conv_state_f32 (single state row, d_conv 4, no bias) - the recurrent state is read in the GDN kernel through the ids tensor - the q/k l2 norms, softplus*A gate and sigmoid(beta) are folded into the GDN kernel (S_v 128, warp 32) - rms_norm * weight * silu(z) -> rms_norm_mul_gated_f32 qwen35, qwen35moe and qwen3next order their graph nodes to match. Other shapes keep the old kernels. tests: GDN decode fusion, RMS_NORM_MUL_GATED. Per GDN layer 27B tensor 58.7 -> 21.3 us; kernels per token 1676 -> 1210. tg128: 27B Q8_0 tensor +8.6%, layer +4.8%, 27B IQ4_XS +3.8%, 4B +7.6%, 35B-A3B tensor +11.5%, Coder-Next 80B tensor +6.5%; prefill flat. test-backend-ops 19011/19011, PPL and greedy output identical. p100-series port (v5.34): fused softplus uses upstream ggml-org#31086's log1pf like op_softplus; the Ampere MMA GDN path is skipped when the inputs are fused; Kane's Pascal chunked-prefill hunk dropped (not carried). Cherry-picked-from: KaneQuah@074fa77 Assisted-by: Claude Opus 5.5

This reverts commit a074215. This fork carries its own DeepSeek-V4.1 implementation (deepseek41, added back later in this series) with the hot/cold expert split, the fused lightning indexer, the DSpark drafter and the RPC layer split. Upstream's implementation (ggml-org#31086) and its follow-up fixes (ggml-org#31137) conflict with it in the arch tables, the DSV4 cache, the hyper-connection helpers and the converter, so both are reverted first and the fork's version is applied on top. This commit removes the follow-up; the next removes ggml-org#31086. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This reverts commit 74bca39. Upstream's DeepSeek-V4.1 support is replaced by the fork's own implementation (deepseek41, later in this series), which the fork's GGUF files, CUDA kernels and server paths are built for. Upstream changes made after ggml-org#31086 are kept where they are not DeepSeek-V4.1 specific: ggml-org#31142 output structs (build_hc_pre returns hc_pre with the mixes computed inline again), ggml-org#31141 no_alloc and ggml-org#31200 n_pad in the DSV4 cache constructors, ggml-org#31368 rs_max / update_rs_for_ubatches, its tensors and build_dspark_block_mask. tests/llama-context still names LLM_ARCH_DEEPSEEK41, which the fork adds back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The small kernels between the qkv/z and ssm_out mat-vecs of a GDN layer (14 per layer in decode) are now 4: - get_rows + concat + cpy + ssm_conv + silu -> ssm_conv_state_f32 (single state row, d_conv 4, no bias) - the recurrent state is read in the GDN kernel through the ids tensor - the q/k l2 norms, softplus*A gate and sigmoid(beta) are folded into the GDN kernel (S_v 128, warp 32) - rms_norm * weight * silu(z) -> rms_norm_mul_gated_f32 qwen35, qwen35moe and qwen3next order their graph nodes to match. Other shapes keep the old kernels. tests: GDN decode fusion, RMS_NORM_MUL_GATED. Per GDN layer 27B tensor 58.7 -> 21.3 us; kernels per token 1676 -> 1210. tg128: 27B Q8_0 tensor +8.6%, layer +4.8%, 27B IQ4_XS +3.8%, 4B +7.6%, 35B-A3B tensor +11.5%, Coder-Next 80B tensor +6.5%; prefill flat. test-backend-ops 19011/19011, PPL and greedy output identical. p100-series port (v5.34): fused softplus uses upstream ggml-org#31086's log1pf like op_softplus; the Ampere MMA GDN path is skipped when the inputs are fused; Kane's Pascal chunked-prefill hunk dropped (not carried). Cherry-picked-from: KaneQuah@074fa77 Assisted-by: Claude Opus 5.5

…r, QSA/indexer) Re-based onto a691b23: 9 files / 34 hunks (the probe predicted 7 files; the cascades were llama-memory-recurrent.cpp and tests/CMakeLists.txt). Decision D3 was measured, not assumed -- see below; this commit keeps the DELIVERY's implementation. Base-API drift this rung had to re-derive (all compiler-driven, no behaviour change): - llama-kv-cache: the ctor keeps BOTH indexer flags -- upstream's `is_indexer` (indexer RoPE params) and the delivery's `v_enabled` (keys-only caches) -- and drops the delivery's `n_pad` ctor parameter and member: upstream ggml-org#31200 removed them and hardcodes the 256-cell floor in the size math, so only the derived `n_pad_prefill` (LLAMA_KV_N_PAD_PREFILL / _MIN / _FROM) survives. `get_n_kv()` now REQUIRES n_tokens (no silent default: a 0 would turn the prefill pad off at that call site) and llama-kv-cache-dsv4 passes it. - llama-memory-hybrid-idx: the keys-only indexer cache is built with upstream's post-ggml-org#31141 ctor (no_alloc, no n_pad) plus is_indexer/v_enabled; the pool body is the delivery's. - llama-memory-recurrent: R04 wins (upstream's rs_max bound, the tracker stays dropped) and the block's two real additions are kept -- the getenv-once `rs_rollback_debug()` helper (issue ggml-org#273) and the `has_recr` gate that stops the qwen4exp MTP context (whose recurrent filter matches nothing) spamming the non-consecutive-position warning. - models.h / qwen4exp.cpp: ggml-org#31733 made `model` a member of llm_graph_context, so the graph and graph_mtp ctors lose their `const llama_model &` argument (and the class its shadowing `const llama_model & model` member); ggml-org#31073 turned the rotation constants into per-layer vectors (`self_k_rot[il]`) and ggml-org#29510/ggml-org#31683 replaced `get_kq_mask()` with `self_attn_mask.raw`; build_attn_mha grew `n_kv_max`. - indexer-score.cu: `ggml_rope_yarn_corr_dims` gained the truncate flag. - tests/CMakeLists.txt: R03 had already re-placed every registration this block adds (ggml-org#30880 moved the generated-model fixtures), so only `test-lazy-reader` is new; test-llama-archs keeps upstream's get_model/get_model_and_ctx split with the block's `lazy_buf_size` handed over as LLAMA_LAZY_BUF_MB, and test-backend-ops keeps BOTH sides' Flash-Attention classes (upstream's kv_rows and the delivery's QSA) and both perf blocks. D3 (qwen4exp / indexer / sparse selection) -- MEASURED, decision: take the delivery's implementation. Upstream now carries its own qwen4exp (1514 lines, by upstream contributors: ggml-org#30965, ggml-org#31243, ggml-org#31443, ggml-org#30927, ggml-org#31086) with its own HC mix/combine (plain ops), QSA selection (build_qsa_sel, an additive mask) and indexer (plain ops); the delivery's is 3197 lines with fused HC ops (HC_MIX/HC_COMBINE, HC16 planar layout), the QSA store+top-k selection (no mask), INDEXER_SCORE/TOPK/FILL and a keys-only indexer cache. A/B on the same GGUFs, same build: Flash-Next GSQ-IQ3_XXS, 3 GPU -sm tensor, -c 8192, 5256-token prompt upstream pp 1399.7 t/s tg 53.5 t/s text 7a8734a9f1a6 delivery pp 1432.1 t/s tg 70.1 t/s text 7a8734a9f1a6 <-- identical Qwen3.8-27B UD-Q4_K_M, 1 GPU, same prompt upstream pp 1311.1 t/s tg 32.4 t/s text b905c9763a58 delivery pp 1303.2 t/s tg 32.4 t/s text b21e94a5b6e1 (neutral; text diverges, needs the KLD gate to judge) So on the config this repo actually validates the delivery's implementation is BIT-IDENTICAL to upstream's and +31 % decode (+2.3 % prefill); on the 27B it is throughput-neutral and its greedy text diverges at 16 tokens (not a purity claim either way -- the chain differs, and the R17 KLD gate is the oracle). Keeping the delivery's implementation also keeps r235's qwen4exp validation and the two delivery gates (gate-hc-fusions.sh, gate-qwen4exp-quant-coherence.sh) meaningful. Validated: build green, no `<<<<<<<`; dense 4B (1472ebdadd46) and Qwen3-Coder-30B-A3B (013d09f165a5) unchanged; Flash-Next and 27B run to completion on the VMM build; gate-hc-fusions.sh PASS (5/5 hard arms byte-identical to the chain, 449 chars d62340fbf566); gate-qwen4exp-quant-coherence.sh PASS (IQ3_XXS / IQ4_NL / Q4_K_M: gather on == off, no '////'). Known drift to record at R17: on this tree the two REPORTED dsv4_pre/post arms no longer agree with the chain on the BF16-hc class (chain 449 chars d62340fbf566, dsv4 arms 6a102dbc62d6 = the value the chain produced at r235), i.e. the HC chain itself moved with the re-base while the fused pre/post ops still reproduce the old one. The gate reports (does not fail) them, and its header says byte-identity IS expected on this class -- the soft-arm expectation needs a decision.

This reverts commit a074215. This fork carries its own DeepSeek-V4.1 implementation (deepseek41, added back later in this series) with the hot/cold expert split, the fused lightning indexer, the DSpark drafter and the RPC layer split. Upstream's implementation (ggml-org#31086) and its follow-up fixes (ggml-org#31137) conflict with it in the arch tables, the DSV4 cache, the hyper-connection helpers and the converter, so both are reverted first and the fork's version is applied on top. This commit removes the follow-up; the next removes ggml-org#31086. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This reverts commit 74bca39. Upstream's DeepSeek-V4.1 support is replaced by the fork's own implementation (deepseek41, later in this series), which the fork's GGUF files, CUDA kernels and server paths are built for. Upstream changes made after ggml-org#31086 are kept where they are not DeepSeek-V4.1 specific: ggml-org#31142 output structs (build_hc_pre returns hc_pre with the mixes computed inline again), ggml-org#31141 no_alloc and ggml-org#31200 n_pad in the DSV4 cache constructors, ggml-org#31368 rs_max / update_rs_for_ubatches, its tensors and build_dspark_block_mask. tests/llama-context still names LLM_ARCH_DEEPSEEK41, which the fork adds back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…r, QSA/indexer) Re-based onto a691b23: 9 files / 34 hunks (the probe predicted 7 files; the cascades were llama-memory-recurrent.cpp and tests/CMakeLists.txt). Decision D3 was measured, not assumed -- see below; this commit keeps the DELIVERY's implementation. Base-API drift this rung had to re-derive (all compiler-driven, no behaviour change): - llama-kv-cache: the ctor keeps BOTH indexer flags -- upstream's `is_indexer` (indexer RoPE params) and the delivery's `v_enabled` (keys-only caches) -- and drops the delivery's `n_pad` ctor parameter and member: upstream ggml-org#31200 removed them and hardcodes the 256-cell floor in the size math, so only the derived `n_pad_prefill` (LLAMA_KV_N_PAD_PREFILL / _MIN / _FROM) survives. `get_n_kv()` now REQUIRES n_tokens (no silent default: a 0 would turn the prefill pad off at that call site) and llama-kv-cache-dsv4 passes it. - llama-memory-hybrid-idx: the keys-only indexer cache is built with upstream's post-ggml-org#31141 ctor (no_alloc, no n_pad) plus is_indexer/v_enabled; the pool body is the delivery's. - llama-memory-recurrent: R04 wins (upstream's rs_max bound, the tracker stays dropped) and the block's two real additions are kept -- the getenv-once `rs_rollback_debug()` helper (issue ggml-org#273) and the `has_recr` gate that stops the qwen4exp MTP context (whose recurrent filter matches nothing) spamming the non-consecutive-position warning. - models.h / qwen4exp.cpp: ggml-org#31733 made `model` a member of llm_graph_context, so the graph and graph_mtp ctors lose their `const llama_model &` argument (and the class its shadowing `const llama_model & model` member); ggml-org#31073 turned the rotation constants into per-layer vectors (`self_k_rot[il]`) and ggml-org#29510/ggml-org#31683 replaced `get_kq_mask()` with `self_attn_mask.raw`; build_attn_mha grew `n_kv_max`. - indexer-score.cu: `ggml_rope_yarn_corr_dims` gained the truncate flag. - tests/CMakeLists.txt: R03 had already re-placed every registration this block adds (ggml-org#30880 moved the generated-model fixtures), so only `test-lazy-reader` is new; test-llama-archs keeps upstream's get_model/get_model_and_ctx split with the block's `lazy_buf_size` handed over as LLAMA_LAZY_BUF_MB, and test-backend-ops keeps BOTH sides' Flash-Attention classes (upstream's kv_rows and the delivery's QSA) and both perf blocks. D3 (qwen4exp / indexer / sparse selection) -- MEASURED, decision: take the delivery's implementation. Upstream now carries its own qwen4exp (1514 lines, by upstream contributors: ggml-org#30965, ggml-org#31243, ggml-org#31443, ggml-org#30927, ggml-org#31086) with its own HC mix/combine (plain ops), QSA selection (build_qsa_sel, an additive mask) and indexer (plain ops); the delivery's is 3197 lines with fused HC ops (HC_MIX/HC_COMBINE, HC16 planar layout), the QSA store+top-k selection (no mask), INDEXER_SCORE/TOPK/FILL and a keys-only indexer cache. A/B on the same GGUFs, same build: Flash-Next GSQ-IQ3_XXS, 3 GPU -sm tensor, -c 8192, 5256-token prompt upstream pp 1399.7 t/s tg 53.5 t/s text 7a8734a9f1a6 delivery pp 1432.1 t/s tg 70.1 t/s text 7a8734a9f1a6 <-- identical Qwen3.8-27B UD-Q4_K_M, 1 GPU, same prompt upstream pp 1311.1 t/s tg 32.4 t/s text b905c9763a58 delivery pp 1303.2 t/s tg 32.4 t/s text b21e94a5b6e1 (neutral; text diverges, needs the KLD gate to judge) So on the config this repo actually validates the delivery's implementation is BIT-IDENTICAL to upstream's and +31 % decode (+2.3 % prefill); on the 27B it is throughput-neutral and its greedy text diverges at 16 tokens (not a purity claim either way -- the chain differs, and the R17 KLD gate is the oracle). Keeping the delivery's implementation also keeps r235's qwen4exp validation and the two delivery gates (gate-hc-fusions.sh, gate-qwen4exp-quant-coherence.sh) meaningful. Validated: build green, no `<<<<<<<`; dense 4B (1472ebdadd46) and Qwen3-Coder-30B-A3B (013d09f165a5) unchanged; Flash-Next and 27B run to completion on the VMM build; gate-hc-fusions.sh PASS (5/5 hard arms byte-identical to the chain, 449 chars d62340fbf566); gate-qwen4exp-quant-coherence.sh PASS (IQ3_XXS / IQ4_NL / Q4_K_M: gather on == off, no '////'). Known drift to record at R17: on this tree the two REPORTED dsv4_pre/post arms no longer agree with the chain on the BF16-hc class (chain 449 chars d62340fbf566, dsv4 arms 6a102dbc62d6 = the value the chain produced at r235), i.e. the HC chain itself moved with the re-base while the fused pre/post ops still reproduce the old one. The gate reports (does not fail) them, and its header says byte-identity IS expected on this class -- the soft-arm expectation needs a decision.

…r, QSA/indexer) Re-based onto a691b23: 9 files / 34 hunks (the probe predicted 7 files; the cascades were llama-memory-recurrent.cpp and tests/CMakeLists.txt). Decision D3 was measured, not assumed -- see below; this commit keeps the DELIVERY's implementation. Base-API drift this rung had to re-derive (all compiler-driven, no behaviour change): - llama-kv-cache: the ctor keeps BOTH indexer flags -- upstream's `is_indexer` (indexer RoPE params) and the delivery's `v_enabled` (keys-only caches) -- and drops the delivery's `n_pad` ctor parameter and member: upstream ggml-org#31200 removed them and hardcodes the 256-cell floor in the size math, so only the derived `n_pad_prefill` (LLAMA_KV_N_PAD_PREFILL / _MIN / _FROM) survives. `get_n_kv()` now REQUIRES n_tokens (no silent default: a 0 would turn the prefill pad off at that call site) and llama-kv-cache-dsv4 passes it. - llama-memory-hybrid-idx: the keys-only indexer cache is built with upstream's post-ggml-org#31141 ctor (no_alloc, no n_pad) plus is_indexer/v_enabled; the pool body is the delivery's. - llama-memory-recurrent: R04 wins (upstream's rs_max bound, the tracker stays dropped) and the block's two real additions are kept -- the getenv-once `rs_rollback_debug()` helper (issue ggml-org#273) and the `has_recr` gate that stops the qwen4exp MTP context (whose recurrent filter matches nothing) spamming the non-consecutive-position warning. - models.h / qwen4exp.cpp: ggml-org#31733 made `model` a member of llm_graph_context, so the graph and graph_mtp ctors lose their `const llama_model &` argument (and the class its shadowing `const llama_model & model` member); ggml-org#31073 turned the rotation constants into per-layer vectors (`self_k_rot[il]`) and ggml-org#29510/ggml-org#31683 replaced `get_kq_mask()` with `self_attn_mask.raw`; build_attn_mha grew `n_kv_max`. - indexer-score.cu: `ggml_rope_yarn_corr_dims` gained the truncate flag. - tests/CMakeLists.txt: R03 had already re-placed every registration this block adds (ggml-org#30880 moved the generated-model fixtures), so only `test-lazy-reader` is new; test-llama-archs keeps upstream's get_model/get_model_and_ctx split with the block's `lazy_buf_size` handed over as LLAMA_LAZY_BUF_MB, and test-backend-ops keeps BOTH sides' Flash-Attention classes (upstream's kv_rows and the delivery's QSA) and both perf blocks. D3 (qwen4exp / indexer / sparse selection) -- MEASURED, decision: take the delivery's implementation. Upstream now carries its own qwen4exp (1514 lines, by upstream contributors: ggml-org#30965, ggml-org#31243, ggml-org#31443, ggml-org#30927, ggml-org#31086) with its own HC mix/combine (plain ops), QSA selection (build_qsa_sel, an additive mask) and indexer (plain ops); the delivery's is 3197 lines with fused HC ops (HC_MIX/HC_COMBINE, HC16 planar layout), the QSA store+top-k selection (no mask), INDEXER_SCORE/TOPK/FILL and a keys-only indexer cache. A/B on the same GGUFs, same build: Flash-Next GSQ-IQ3_XXS, 3 GPU -sm tensor, -c 8192, 5256-token prompt upstream pp 1399.7 t/s tg 53.5 t/s text 7a8734a9f1a6 delivery pp 1432.1 t/s tg 70.1 t/s text 7a8734a9f1a6 <-- identical Qwen3.8-27B UD-Q4_K_M, 1 GPU, same prompt upstream pp 1311.1 t/s tg 32.4 t/s text b905c9763a58 delivery pp 1303.2 t/s tg 32.4 t/s text b21e94a5b6e1 (neutral; text diverges, needs the KLD gate to judge) So on the config this repo actually validates the delivery's implementation is BIT-IDENTICAL to upstream's and +31 % decode (+2.3 % prefill); on the 27B it is throughput-neutral and its greedy text diverges at 16 tokens (not a purity claim either way -- the chain differs, and the R17 KLD gate is the oracle). Keeping the delivery's implementation also keeps r235's qwen4exp validation and the two delivery gates (gate-hc-fusions.sh, gate-qwen4exp-quant-coherence.sh) meaningful. Validated: build green, no `<<<<<<<`; dense 4B (1472ebdadd46) and Qwen3-Coder-30B-A3B (013d09f165a5) unchanged; Flash-Next and 27B run to completion on the VMM build; gate-hc-fusions.sh PASS (5/5 hard arms byte-identical to the chain, 449 chars d62340fbf566); gate-qwen4exp-quant-coherence.sh PASS (IQ3_XXS / IQ4_NL / Q4_K_M: gather on == off, no '////'). Known drift to record at R17: on this tree the two REPORTED dsv4_pre/post arms no longer agree with the chain on the BF16-hc class (chain 449 chars d62340fbf566, dsv4 arms 6a102dbc62d6 = the value the chain produced at r235), i.e. the HC chain itself moved with the re-base while the fused pre/post ops still reproduce the old one. The gate reports (does not fail) them, and its header says byte-identity IS expected on this class -- the soft-arm expectation needs a decision. r242 (contributor PR ggml-org#288, bsp2020): the qwen4exp PREFILL chain is fused and its BF16 GEMM activations are produced, not converted. - `DSV4_HC_POST -> RMS_NORM -> MUL(gamma)` (only emitted under a Meta device, `fused_dsv4_hc_post`) becomes one launch, `dsv4_hc_post_rms_f32` in norm.cu: the hc_post sum with `dsv4_hc_post_f32`'s own two statements, stored (it is the next layer's residual), then `rms_norm_f32`'s SCALAR path (1024-thread, tid-strided) reduction, scale and gamma product -- bit-identical, not rounding-level. Matcher in ggml-cuda-try-fuse.cu (its range check uses the tensor span and allows only the hc_post/residual in-place pair; the window is not contiguous, so the shared check does not apply). ggml-cuda.cu's graph optimizer keeps x/residual/post alive until the MUL. `GGML_CUDA_HC_POST_RMS=0`; census `GGML_CUDA_HC_POST_RMS_DEBUG=1` (INFO: 1302 fused=1, 0 declined on a 31.5k prefill -- a gate on a silent fusion is vacuous, INVARIANTS r225/r238). - `dsv4_hc_post_rms`'s xn and `dsv4_hc_pre`'s mixed stream write the RNE BF16 copy of their output in the same pass and leave it in `bf16_src1_cache`, so the rocBLAS BF16 hc GEMMs skip their F32 -> BF16 conversion. `GGML_CUDA_BF16_PRODUCER=0` off, `=2` re-converts every hit and compares (diagnostic only). Validated (2 R9700, `-sm tensor`, Flash-Next GSQ-RCO IQ3_S): prefill 2330 -> 2470 t/s (+6.0 %, 31.5k prompt), 2421 -> 2617 (+8.1 %, 101k), +6.8 % on the delivery-style arm; greedy text identical in every arm and on the MTP decode arm; prefill-logit KLD -0.000000 / 100.000 %; gate-hc-fusions.sh PASS; HC op oracles 30/30; gfx1100 and gfx1151 text identical with the fusion firing (372 fused, 0 declined); 1 device neutral. Record: archive/work/hc-post-rms-bf16/, WORKLOG 261109.007, INVARIANTS r242.

…r, QSA/indexer) Re-based onto a691b23: 9 files / 34 hunks (the probe predicted 7 files; the cascades were llama-memory-recurrent.cpp and tests/CMakeLists.txt). Decision D3 was measured, not assumed -- see below; this commit keeps the DELIVERY's implementation. Base-API drift this rung had to re-derive (all compiler-driven, no behaviour change): - llama-kv-cache: the ctor keeps BOTH indexer flags -- upstream's `is_indexer` (indexer RoPE params) and the delivery's `v_enabled` (keys-only caches) -- and drops the delivery's `n_pad` ctor parameter and member: upstream ggml-org#31200 removed them and hardcodes the 256-cell floor in the size math, so only the derived `n_pad_prefill` (LLAMA_KV_N_PAD_PREFILL / _MIN / _FROM) survives. `get_n_kv()` now REQUIRES n_tokens (no silent default: a 0 would turn the prefill pad off at that call site) and llama-kv-cache-dsv4 passes it. - llama-memory-hybrid-idx: the keys-only indexer cache is built with upstream's post-ggml-org#31141 ctor (no_alloc, no n_pad) plus is_indexer/v_enabled; the pool body is the delivery's. - llama-memory-recurrent: R04 wins (upstream's rs_max bound, the tracker stays dropped) and the block's two real additions are kept -- the getenv-once `rs_rollback_debug()` helper (issue ggml-org#273) and the `has_recr` gate that stops the qwen4exp MTP context (whose recurrent filter matches nothing) spamming the non-consecutive-position warning. - models.h / qwen4exp.cpp: ggml-org#31733 made `model` a member of llm_graph_context, so the graph and graph_mtp ctors lose their `const llama_model &` argument (and the class its shadowing `const llama_model & model` member); ggml-org#31073 turned the rotation constants into per-layer vectors (`self_k_rot[il]`) and ggml-org#29510/ggml-org#31683 replaced `get_kq_mask()` with `self_attn_mask.raw`; build_attn_mha grew `n_kv_max`. - indexer-score.cu: `ggml_rope_yarn_corr_dims` gained the truncate flag. - tests/CMakeLists.txt: R03 had already re-placed every registration this block adds (ggml-org#30880 moved the generated-model fixtures), so only `test-lazy-reader` is new; test-llama-archs keeps upstream's get_model/get_model_and_ctx split with the block's `lazy_buf_size` handed over as LLAMA_LAZY_BUF_MB, and test-backend-ops keeps BOTH sides' Flash-Attention classes (upstream's kv_rows and the delivery's QSA) and both perf blocks. D3 (qwen4exp / indexer / sparse selection) -- MEASURED, decision: take the delivery's implementation. Upstream now carries its own qwen4exp (1514 lines, by upstream contributors: ggml-org#30965, ggml-org#31243, ggml-org#31443, ggml-org#30927, ggml-org#31086) with its own HC mix/combine (plain ops), QSA selection (build_qsa_sel, an additive mask) and indexer (plain ops); the delivery's is 3197 lines with fused HC ops (HC_MIX/HC_COMBINE, HC16 planar layout), the QSA store+top-k selection (no mask), INDEXER_SCORE/TOPK/FILL and a keys-only indexer cache. A/B on the same GGUFs, same build: Flash-Next GSQ-IQ3_XXS, 3 GPU -sm tensor, -c 8192, 5256-token prompt upstream pp 1399.7 t/s tg 53.5 t/s text 7a8734a9f1a6 delivery pp 1432.1 t/s tg 70.1 t/s text 7a8734a9f1a6 <-- identical Qwen3.8-27B UD-Q4_K_M, 1 GPU, same prompt upstream pp 1311.1 t/s tg 32.4 t/s text b905c9763a58 delivery pp 1303.2 t/s tg 32.4 t/s text b21e94a5b6e1 (neutral; text diverges, needs the KLD gate to judge) So on the config this repo actually validates the delivery's implementation is BIT-IDENTICAL to upstream's and +31 % decode (+2.3 % prefill); on the 27B it is throughput-neutral and its greedy text diverges at 16 tokens (not a purity claim either way -- the chain differs, and the R17 KLD gate is the oracle). Keeping the delivery's implementation also keeps r235's qwen4exp validation and the two delivery gates (gate-hc-fusions.sh, gate-qwen4exp-quant-coherence.sh) meaningful. Validated: build green, no `<<<<<<<`; dense 4B (1472ebdadd46) and Qwen3-Coder-30B-A3B (013d09f165a5) unchanged; Flash-Next and 27B run to completion on the VMM build; gate-hc-fusions.sh PASS (5/5 hard arms byte-identical to the chain, 449 chars d62340fbf566); gate-qwen4exp-quant-coherence.sh PASS (IQ3_XXS / IQ4_NL / Q4_K_M: gather on == off, no '////'). Known drift to record at R17: on this tree the two REPORTED dsv4_pre/post arms no longer agree with the chain on the BF16-hc class (chain 449 chars d62340fbf566, dsv4 arms 6a102dbc62d6 = the value the chain produced at r235), i.e. the HC chain itself moved with the re-base while the fused pre/post ops still reproduce the old one. The gate reports (does not fail) them, and its header says byte-identity IS expected on this class -- the soft-arm expectation needs a decision. r242 (contributor PR ggml-org#288, bsp2020): the qwen4exp PREFILL chain is fused and its BF16 GEMM activations are produced, not converted. - `DSV4_HC_POST -> RMS_NORM -> MUL(gamma)` (only emitted under a Meta device, `fused_dsv4_hc_post`) becomes one launch, `dsv4_hc_post_rms_f32` in norm.cu: the hc_post sum with `dsv4_hc_post_f32`'s own two statements, stored (it is the next layer's residual), then `rms_norm_f32`'s SCALAR path (1024-thread, tid-strided) reduction, scale and gamma product -- bit-identical, not rounding-level. Matcher in ggml-cuda-try-fuse.cu (its range check uses the tensor span and allows only the hc_post/residual in-place pair; the window is not contiguous, so the shared check does not apply). ggml-cuda.cu's graph optimizer keeps x/residual/post alive until the MUL. `GGML_CUDA_HC_POST_RMS=0`; census `GGML_CUDA_HC_POST_RMS_DEBUG=1` (INFO: 1302 fused=1, 0 declined on a 31.5k prefill -- a gate on a silent fusion is vacuous, INVARIANTS r225/r238). - `dsv4_hc_post_rms`'s xn and `dsv4_hc_pre`'s mixed stream write the RNE BF16 copy of their output in the same pass and leave it in `bf16_src1_cache`, so the rocBLAS BF16 hc GEMMs skip their F32 -> BF16 conversion. `GGML_CUDA_BF16_PRODUCER=0` off, `=2` re-converts every hit and compares (diagnostic only). Validated (2 R9700, `-sm tensor`, Flash-Next GSQ-RCO IQ3_S): prefill 2330 -> 2470 t/s (+6.0 %, 31.5k prompt), 2421 -> 2617 (+8.1 %, 101k), +6.8 % on the delivery-style arm; greedy text identical in every arm and on the MTP decode arm; prefill-logit KLD -0.000000 / 100.000 %; gate-hc-fusions.sh PASS; HC op oracles 30/30; gfx1100 and gfx1151 text identical with the fusion firing (372 fused, 0 declined); 1 device neutral. Record: archive/work/hc-post-rms-bf16/, WORKLOG 261109.007, INVARIANTS r242. r243 (contributor PR ggml-org#289, bsp2020): the host PLE gather fans out over --threads and writes the graph input in place. - `llm_graph_input_ple::set_input`: the rows of `per_layer_token_embd` are dequantized straight into `emb->data` when the input tensor lives in a host buffer, so the whole-input host memcpy per prompt ubatch is gone (the buffer is resolved the way `ggml_backend_tensor_set()` resolves it, view_src first, so a view cannot be mis-detected; a device, split or RPC buffer keeps the scratch + `ggml_backend_tensor_set` path). `LLAMA_PLE_DIRECT=0` restores it. - The gather is split over one thread per 4096 rows, so fewer than 8192 rows (decode) stays on the calling thread; the width is the smallest of 8, `std::thread::hardware_concurrency()` and `cparams.n_threads_batch` (i.e. `--threads`/`-tb`, which is what decides it), with `LLAMA_PLE_THREADS` overriding the ceiling for the A/B. Each row is dequantized on its own with the same `to_float`, so the bytes are identical; a worker that cannot be started leaves its rows to the caller (the ranges cover every row whether or not all of them start). ggml's threadpool only ever executes a `ggml_graph_compute`, and it is idle here (the GPUs run the prefill), so the fan-out spawns its own threads exactly as `llama_lazy_reader::gather()` and `llama_state_copy_jobs` already do. Census `LLAMA_PLE_DEBUG=1` (INFO: `ple gather: <rows> rows, <threads> threads, direct=1`), because a silent optimisation makes a later arm vacuous. - Measured (2x R9700 `0,2`, `-sm tensor`, Flash-Next GSQ-RCO IQ3_S, 31.5k prompt, `-ncmoe 47 -lme 47`, `--load-mode mmap`): the delivery shape (`-ub 4096` + attendance pool) **2338.5 -> 2369.2 t/s (+1.31 %)**, 6 runs per side with no overlap between the arms (and +0.77 % again with the 7-thread ceiling `-t 7` sets); `-ub 4096` without the pool 2286.8 -> 2313.3 (+1.16 %); the contributor's own `-ub 6144` field shape 2384.3 -> 2385.7 (+0.06 %) and its 101k prompt 2444.1 -> 2451.9 (+0.32 %) -- the win is `-ub 4096`-shaped (more ubatch boundaries), not a field-shape one. Isolated on one binary with both switches off as the control: threads alone +0.69 %, direct write alone +0.17 %. `llama-batched-bench -npl 1,4,8 -npp 512 -ntg 32`: PP 592.5 -> 597.2 / 2226.0 -> 2242.9 / 2465.0 -> 2539.8 t/s, decode 56.0/160.3/237.1 -> 56.2/157.5/240.7 (inside the spread). - Purity: greedy text identical in every arm (31.5k `9de3ae5502bc`; `draft-mtp n3` at -n 256 `e6ecbe2cd1c8`; 1 device `-sm layer` `0f5005a4398f`; a Qwen3.5-4B control `83eec5e9b4f0`); prefill-logit KLD **-0.000000 / 100.000 %** (`--sm tensor --device 0,2`, base recorded from the r242 build with the same arm) -- the logits are unchanged, not merely inside tolerance; no GPU faults over every run (`check-gpu-faults.sh`). - The contributor's motivation overstated the work per ubatch (~300k rows and a 200 MB copy at `-ub 4096`; `ple_n_heads` is 16, so it is 65 536 rows / 41.9 MB, and 98 304 / 62.9 MB at `-ub 6144`), and their field-shape deltas do not reproduce here; their `-ub 4096` numbers do, and that is the delivery's own shape. Record: archive/work/ple-gather/, WORKLOG 261109.008, INVARIANTS r243, ENVIRONMENT.md.

This reverts commit a074215. This fork carries its own DeepSeek-V4.1 implementation (deepseek41, added back later in this series) with the hot/cold expert split, the fused lightning indexer, the DSpark drafter and the RPC layer split. Upstream's implementation (ggml-org#31086) and its follow-up fixes (ggml-org#31137) conflict with it in the arch tables, the DSV4 cache, the hyper-connection helpers and the converter, so both are reverted first and the fork's version is applied on top. This commit removes the follow-up; the next removes ggml-org#31086. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This reverts commit 74bca39. Upstream's DeepSeek-V4.1 support is replaced by the fork's own implementation (deepseek41, later in this series), which the fork's GGUF files, CUDA kernels and server paths are built for. Upstream changes made after ggml-org#31086 are kept where they are not DeepSeek-V4.1 specific: ggml-org#31142 output structs (build_hc_pre returns hc_pre with the mixes computed inline again), ggml-org#31141 no_alloc and ggml-org#31200 n_pad in the DSV4 cache constructors, ggml-org#31368 rs_max / update_rs_for_ubatches, its tensors and build_dspark_block_mask. tests/llama-context still names LLM_ARCH_DEEPSEEK41, which the fork adds back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…r, QSA/indexer) Re-based onto a691b23: 9 files / 34 hunks (the probe predicted 7 files; the cascades were llama-memory-recurrent.cpp and tests/CMakeLists.txt). Decision D3 was measured, not assumed -- see below; this commit keeps the DELIVERY's implementation. Base-API drift this rung had to re-derive (all compiler-driven, no behaviour change): - llama-kv-cache: the ctor keeps BOTH indexer flags -- upstream's `is_indexer` (indexer RoPE params) and the delivery's `v_enabled` (keys-only caches) -- and drops the delivery's `n_pad` ctor parameter and member: upstream ggml-org#31200 removed them and hardcodes the 256-cell floor in the size math, so only the derived `n_pad_prefill` (LLAMA_KV_N_PAD_PREFILL / _MIN / _FROM) survives. `get_n_kv()` now REQUIRES n_tokens (no silent default: a 0 would turn the prefill pad off at that call site) and llama-kv-cache-dsv4 passes it. - llama-memory-hybrid-idx: the keys-only indexer cache is built with upstream's post-ggml-org#31141 ctor (no_alloc, no n_pad) plus is_indexer/v_enabled; the pool body is the delivery's. - llama-memory-recurrent: R04 wins (upstream's rs_max bound, the tracker stays dropped) and the block's two real additions are kept -- the getenv-once `rs_rollback_debug()` helper (issue ggml-org#273) and the `has_recr` gate that stops the qwen4exp MTP context (whose recurrent filter matches nothing) spamming the non-consecutive-position warning. - models.h / qwen4exp.cpp: ggml-org#31733 made `model` a member of llm_graph_context, so the graph and graph_mtp ctors lose their `const llama_model &` argument (and the class its shadowing `const llama_model & model` member); ggml-org#31073 turned the rotation constants into per-layer vectors (`self_k_rot[il]`) and ggml-org#29510/ggml-org#31683 replaced `get_kq_mask()` with `self_attn_mask.raw`; build_attn_mha grew `n_kv_max`. - indexer-score.cu: `ggml_rope_yarn_corr_dims` gained the truncate flag. - tests/CMakeLists.txt: R03 had already re-placed every registration this block adds (ggml-org#30880 moved the generated-model fixtures), so only `test-lazy-reader` is new; test-llama-archs keeps upstream's get_model/get_model_and_ctx split with the block's `lazy_buf_size` handed over as LLAMA_LAZY_BUF_MB, and test-backend-ops keeps BOTH sides' Flash-Attention classes (upstream's kv_rows and the delivery's QSA) and both perf blocks. D3 (qwen4exp / indexer / sparse selection) -- MEASURED, decision: take the delivery's implementation. Upstream now carries its own qwen4exp (1514 lines, by upstream contributors: ggml-org#30965, ggml-org#31243, ggml-org#31443, ggml-org#30927, ggml-org#31086) with its own HC mix/combine (plain ops), QSA selection (build_qsa_sel, an additive mask) and indexer (plain ops); the delivery's is 3197 lines with fused HC ops (HC_MIX/HC_COMBINE, HC16 planar layout), the QSA store+top-k selection (no mask), INDEXER_SCORE/TOPK/FILL and a keys-only indexer cache. A/B on the same GGUFs, same build: Flash-Next GSQ-IQ3_XXS, 3 GPU -sm tensor, -c 8192, 5256-token prompt upstream pp 1399.7 t/s tg 53.5 t/s text 7a8734a9f1a6 delivery pp 1432.1 t/s tg 70.1 t/s text 7a8734a9f1a6 <-- identical Qwen3.8-27B UD-Q4_K_M, 1 GPU, same prompt upstream pp 1311.1 t/s tg 32.4 t/s text b905c9763a58 delivery pp 1303.2 t/s tg 32.4 t/s text b21e94a5b6e1 (neutral; text diverges, needs the KLD gate to judge) So on the config this repo actually validates the delivery's implementation is BIT-IDENTICAL to upstream's and +31 % decode (+2.3 % prefill); on the 27B it is throughput-neutral and its greedy text diverges at 16 tokens (not a purity claim either way -- the chain differs, and the R17 KLD gate is the oracle). Keeping the delivery's implementation also keeps r235's qwen4exp validation and the two delivery gates (gate-hc-fusions.sh, gate-qwen4exp-quant-coherence.sh) meaningful. Validated: build green, no `<<<<<<<`; dense 4B (1472ebdadd46) and Qwen3-Coder-30B-A3B (013d09f165a5) unchanged; Flash-Next and 27B run to completion on the VMM build; gate-hc-fusions.sh PASS (5/5 hard arms byte-identical to the chain, 449 chars d62340fbf566); gate-qwen4exp-quant-coherence.sh PASS (IQ3_XXS / IQ4_NL / Q4_K_M: gather on == off, no '////'). Known drift to record at R17: on this tree the two REPORTED dsv4_pre/post arms no longer agree with the chain on the BF16-hc class (chain 449 chars d62340fbf566, dsv4 arms 6a102dbc62d6 = the value the chain produced at r235), i.e. the HC chain itself moved with the re-base while the fused pre/post ops still reproduce the old one. The gate reports (does not fail) them, and its header says byte-identity IS expected on this class -- the soft-arm expectation needs a decision. r242 (contributor PR ggml-org#288, bsp2020): the qwen4exp PREFILL chain is fused and its BF16 GEMM activations are produced, not converted. - `DSV4_HC_POST -> RMS_NORM -> MUL(gamma)` (only emitted under a Meta device, `fused_dsv4_hc_post`) becomes one launch, `dsv4_hc_post_rms_f32` in norm.cu: the hc_post sum with `dsv4_hc_post_f32`'s own two statements, stored (it is the next layer's residual), then `rms_norm_f32`'s SCALAR path (1024-thread, tid-strided) reduction, scale and gamma product -- bit-identical, not rounding-level. Matcher in ggml-cuda-try-fuse.cu (its range check uses the tensor span and allows only the hc_post/residual in-place pair; the window is not contiguous, so the shared check does not apply). ggml-cuda.cu's graph optimizer keeps x/residual/post alive until the MUL. `GGML_CUDA_HC_POST_RMS=0`; census `GGML_CUDA_HC_POST_RMS_DEBUG=1` (INFO: 1302 fused=1, 0 declined on a 31.5k prefill -- a gate on a silent fusion is vacuous, INVARIANTS r225/r238). - `dsv4_hc_post_rms`'s xn and `dsv4_hc_pre`'s mixed stream write the RNE BF16 copy of their output in the same pass and leave it in `bf16_src1_cache`, so the rocBLAS BF16 hc GEMMs skip their F32 -> BF16 conversion. `GGML_CUDA_BF16_PRODUCER=0` off, `=2` re-converts every hit and compares (diagnostic only). Validated (2 R9700, `-sm tensor`, Flash-Next GSQ-RCO IQ3_S): prefill 2330 -> 2470 t/s (+6.0 %, 31.5k prompt), 2421 -> 2617 (+8.1 %, 101k), +6.8 % on the delivery-style arm; greedy text identical in every arm and on the MTP decode arm; prefill-logit KLD -0.000000 / 100.000 %; gate-hc-fusions.sh PASS; HC op oracles 30/30; gfx1100 and gfx1151 text identical with the fusion firing (372 fused, 0 declined); 1 device neutral. Record: archive/work/hc-post-rms-bf16/, WORKLOG 261109.007, INVARIANTS r242. r243 (contributor PR ggml-org#289, bsp2020): the host PLE gather fans out over --threads and writes the graph input in place. - `llm_graph_input_ple::set_input`: the rows of `per_layer_token_embd` are dequantized straight into `emb->data` when the input tensor lives in a host buffer, so the whole-input host memcpy per prompt ubatch is gone (the buffer is resolved the way `ggml_backend_tensor_set()` resolves it, view_src first, so a view cannot be mis-detected; a device, split or RPC buffer keeps the scratch + `ggml_backend_tensor_set` path). `LLAMA_PLE_DIRECT=0` restores it. - The gather is split over one thread per 4096 rows, so fewer than 8192 rows (decode) stays on the calling thread; the width is the smallest of 8, `std::thread::hardware_concurrency()` and `cparams.n_threads_batch` (i.e. `--threads`/`-tb`, which is what decides it), with `LLAMA_PLE_THREADS` overriding the ceiling for the A/B. Each row is dequantized on its own with the same `to_float`, so the bytes are identical; a worker that cannot be started leaves its rows to the caller (the ranges cover every row whether or not all of them start). ggml's threadpool only ever executes a `ggml_graph_compute`, and it is idle here (the GPUs run the prefill), so the fan-out spawns its own threads exactly as `llama_lazy_reader::gather()` and `llama_state_copy_jobs` already do. Census `LLAMA_PLE_DEBUG=1` (INFO: `ple gather: <rows> rows, <threads> threads, direct=1`), because a silent optimisation makes a later arm vacuous. - Measured (2x R9700 `0,2`, `-sm tensor`, Flash-Next GSQ-RCO IQ3_S, 31.5k prompt, `-ncmoe 47 -lme 47`, `--load-mode mmap`): the delivery shape (`-ub 4096` + attendance pool) **2338.5 -> 2369.2 t/s (+1.31 %)**, 6 runs per side with no overlap between the arms (and +0.77 % again with the 7-thread ceiling `-t 7` sets); `-ub 4096` without the pool 2286.8 -> 2313.3 (+1.16 %); the contributor's own `-ub 6144` field shape 2384.3 -> 2385.7 (+0.06 %) and its 101k prompt 2444.1 -> 2451.9 (+0.32 %) -- the win is `-ub 4096`-shaped (more ubatch boundaries), not a field-shape one. Isolated on one binary with both switches off as the control: threads alone +0.69 %, direct write alone +0.17 %. `llama-batched-bench -npl 1,4,8 -npp 512 -ntg 32`: PP 592.5 -> 597.2 / 2226.0 -> 2242.9 / 2465.0 -> 2539.8 t/s, decode 56.0/160.3/237.1 -> 56.2/157.5/240.7 (inside the spread). - Purity: greedy text identical in every arm (31.5k `9de3ae5502bc`; `draft-mtp n3` at -n 256 `e6ecbe2cd1c8`; 1 device `-sm layer` `0f5005a4398f`; a Qwen3.5-4B control `83eec5e9b4f0`); prefill-logit KLD **-0.000000 / 100.000 %** (`--sm tensor --device 0,2`, base recorded from the r242 build with the same arm) -- the logits are unchanged, not merely inside tolerance; no GPU faults over every run (`check-gpu-faults.sh`). - The contributor's motivation overstated the work per ubatch (~300k rows and a 200 MB copy at `-ub 4096`; `ple_n_heads` is 16, so it is 65 536 rows / 41.9 MB, and 98 304 / 62.9 MB at `-ub 6144`), and their field-shape deltas do not reproduce here; their `-ub 4096` numbers do, and that is the delivery's own shape. Record: archive/work/ple-gather/, WORKLOG 261109.008, INVARIANTS r243, ENVIRONMENT.md.

danielhanchen and others added 2 commits October 24, 2026 17:20

model : add DeepSeek-V4.1-Flash (deepseek41)

9479ff0

deepseek41 : remove MXFP8, convert the Engram table to BF16

40414d1

am17an requested review from a team, CISC, ggerganov and pwilkin as code owners October 24, 2026 10:09

github-actions Bot added model Model specific testing Everything test related ggml changes relating to the ggml tensor library for machine learning mtmd Related to multimodal functionality (video/image/audio) CUDA Related to the CUDA backend conversion labels Oct 24, 2026

am17an commented Oct 24, 2026

View reviewed changes

Comment thread src/models/deepseek4.cpp Resolved

am17an added 4 commits October 24, 2026 18:27

deepseek41 : compute the compressed-KV read layers once in the cache

863314b

deepseek41 : use get_cells() for the shared-history check, drop llama…

c028046

…_kv_cache::seq_pos_tok_le

deepseek41 : gather the Engram rows in-graph like qwen4exp, drop the …

51e0cf6

…host gather

deepseek41 : keep the Engram hash constants in hparams.ple_*, one mod…

1ea5ea2

…ule per Engram layer

am17an added 4 commits October 24, 2026 20:36

deepseek41 : drop the LLAMA_DSV41_KV_MIRROR toggle

65d1b65

deepseek41 : keep il last in build_hc_mixes

ddcb043

deepseek41 : sort the top-k rows before the sparse gather

f6435ae

tests : add deepseek41 to the Metal fusion baseline

2052692

pwilkin approved these changes Oct 24, 2026

View reviewed changes

ngxson approved these changes Oct 25, 2026

View reviewed changes

am17an merged commit 74bca39 into master Oct 25, 2026
40 of 45 checks passed

Repzei mentioned this pull request Oct 25, 2026

7am AI 2026-10-25: OpenAI's Codex Security found a serious hole in Node.js, and the fix is out Repzei/7amai#15

Merged

canonrebel04 mentioned this pull request Nov 2, 2026

sync: merge upstream/master c6fb3b5be + 785322d17 (2026-10-26, 2026-11-01) — stacked on #85 canonrebel04/llama.cpp#86

Open

Tont1no added a commit to Tont1no/atomic-llama-cpp-turboquant that referenced this pull request Nov 3, 2026

WIP: DeepSeek-V4.1-Flash (deepseek41) from upstream ggml-org#31086

601b9e3

nbritton mentioned this pull request Nov 6, 2026

Merge llama.cpp v0.9.0 (b12110) mxxm-t/mx-llama.cpp#23

Open

ServeurpersoCom mentioned this pull request Nov 7, 2026

discussion : ggml_prec updates and formalization #31821

Draft

leehack mentioned this pull request Nov 7, 2026

Move llama.cpp to v0.9.0 and audit its CPU ISA source pair leehack/llamadart-native#175

Merged

naualex2023 mentioned this pull request Nov 8, 2026

llama, gguf-split, quant : robustness fixes for giant-tensor models #31891

Open

Nathanw1014 mentioned this pull request Nov 10, 2026

r40-integration: dsv41 runtime + slot-sampler reuse on strix/master halo-box/strix-llama.cpp#273

Open

Navigation Menu

Repository navigation

model: DSV4.1 flash - #31086

model: DSV4.1 flash#31086
am17an merged 10 commits into
masterfrom
aman/dsv41-no-mxfp8

am17an commented Oct 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

pwilkin commented Oct 24, 2026

Uh oh!

ggml-gh-bot Bot commented Oct 24, 2026 •

edited

Loading

Uh oh!

am17an commented Oct 24, 2026

Uh oh!

pwilkin left a comment

Uh oh!

ngxson left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

am17an commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

Uh oh!

pwilkin commented Oct 24, 2026

Uh oh!

ggml-gh-bot Bot commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

am17an commented Oct 24, 2026

Uh oh!

pwilkin left a comment

Choose a reason for hiding this comment

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

am17an commented Oct 24, 2026 •

edited

Loading

ggml-gh-bot Bot commented Oct 24, 2026 •

edited

Loading