llama: refactoring: move model to class member of llm_graph_context by ngxson · Pull Request #31733 · ggml-org/llama.cpp

ngxson · 2026-11-05T13:42:37Z

Overview

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: yes

* feat : added ricursiveai ripple support * fix : mandatiry SWA, scale through helper, comments back * fix : metal * model : make ricursive RoPE/NoPE explicit and validate the layer pattern Write attention.rope_pattern and read it as optional, falling back to the sliding window pattern so existing GGUFs keep loading. Refuse a sliding_window_pattern that does not cover every layer, a model with no sliding layer, and a gate or value residual present in only some layers. The converter derives layer_types when the config omits it, ignores a stale hybrid_layer_pattern and drops tensors of features the config disables. Assisted-by: Claude Opus 5.5 * model : make ricursive swiglu_clamp per-layer Store swiglu_clamp as a per-layer array, read it with get_key_or_arr so the scalar in existing GGUFs still applies to every layer, and refuse a negative or non-finite value. Add a test-llama-archs fixture with mixed per-layer clamps and no rope_pattern, so the saver roundtrip checks both. Assisted-by: Claude Opus 5.5 * model : use the ricursive per-tensor scales and RoPE parameters Pass the attn_gate and output weight scales to build_lora_mm, take ext_factor and attn_factor from the graph instead of hardcoding them, and copy rope_freq_scale_train to the SWA layers instead of pinning it to 1. Cite the reference lines for the value residual and the NoPE layers. Assisted-by: Claude Opus 5.5 * convert : drop the ricursive chat_template.jinja override The precedence between tokenizer_config.json and chat_template.jinja is to be changed in SpecialVocab for all models instead. Until then a new conversion stores the short tokenizer_config.json template; GGUFs that already carry the tool-aware template are not affected. Assisted-by: Claude Opus 5.5 * fix : remove NVFP4 checkpoint quantization block * match #31733 refactor --------- Co-authored-by: Piotr Wilkin <ilintar@gmail.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>

- sm_60 MUL_MAT_ID kernels: keep the per-expert-bounds ids helper (mmid-bounds.cu) that upstream ggml-org#31047 replaced by tile specs - deepseek41: the graph on upstream ggml-org#31733 (model member, no_build base ctor), the llm_attn_mask raw mask, build_attn_mha's kv_rows argument - llama-context: graph_params initialiser; the DFlash device-copy reader event (ggml-org#31506) is recorded in decode_finish (a launched-ahead decode finishes in llama_decode_commit) - kv-cells: seq_rm_all clears the sequence's seq_bits (ggml-org#29510) - graph slots debug: llm_attn_mask; model saver: DEEPSEEK41 in the DSV4 condition Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…r, QSA/indexer) Re-based onto a691b23: 9 files / 34 hunks (the probe predicted 7 files; the cascades were llama-memory-recurrent.cpp and tests/CMakeLists.txt). Decision D3 was measured, not assumed -- see below; this commit keeps the DELIVERY's implementation. Base-API drift this rung had to re-derive (all compiler-driven, no behaviour change): - llama-kv-cache: the ctor keeps BOTH indexer flags -- upstream's `is_indexer` (indexer RoPE params) and the delivery's `v_enabled` (keys-only caches) -- and drops the delivery's `n_pad` ctor parameter and member: upstream ggml-org#31200 removed them and hardcodes the 256-cell floor in the size math, so only the derived `n_pad_prefill` (LLAMA_KV_N_PAD_PREFILL / _MIN / _FROM) survives. `get_n_kv()` now REQUIRES n_tokens (no silent default: a 0 would turn the prefill pad off at that call site) and llama-kv-cache-dsv4 passes it. - llama-memory-hybrid-idx: the keys-only indexer cache is built with upstream's post-ggml-org#31141 ctor (no_alloc, no n_pad) plus is_indexer/v_enabled; the pool body is the delivery's. - llama-memory-recurrent: R04 wins (upstream's rs_max bound, the tracker stays dropped) and the block's two real additions are kept -- the getenv-once `rs_rollback_debug()` helper (issue ggml-org#273) and the `has_recr` gate that stops the qwen4exp MTP context (whose recurrent filter matches nothing) spamming the non-consecutive-position warning. - models.h / qwen4exp.cpp: ggml-org#31733 made `model` a member of llm_graph_context, so the graph and graph_mtp ctors lose their `const llama_model &` argument (and the class its shadowing `const llama_model & model` member); ggml-org#31073 turned the rotation constants into per-layer vectors (`self_k_rot[il]`) and ggml-org#29510/ggml-org#31683 replaced `get_kq_mask()` with `self_attn_mask.raw`; build_attn_mha grew `n_kv_max`. - indexer-score.cu: `ggml_rope_yarn_corr_dims` gained the truncate flag. - tests/CMakeLists.txt: R03 had already re-placed every registration this block adds (ggml-org#30880 moved the generated-model fixtures), so only `test-lazy-reader` is new; test-llama-archs keeps upstream's get_model/get_model_and_ctx split with the block's `lazy_buf_size` handed over as LLAMA_LAZY_BUF_MB, and test-backend-ops keeps BOTH sides' Flash-Attention classes (upstream's kv_rows and the delivery's QSA) and both perf blocks. D3 (qwen4exp / indexer / sparse selection) -- MEASURED, decision: take the delivery's implementation. Upstream now carries its own qwen4exp (1514 lines, by upstream contributors: ggml-org#30965, ggml-org#31243, ggml-org#31443, ggml-org#30927, ggml-org#31086) with its own HC mix/combine (plain ops), QSA selection (build_qsa_sel, an additive mask) and indexer (plain ops); the delivery's is 3197 lines with fused HC ops (HC_MIX/HC_COMBINE, HC16 planar layout), the QSA store+top-k selection (no mask), INDEXER_SCORE/TOPK/FILL and a keys-only indexer cache. A/B on the same GGUFs, same build: Flash-Next GSQ-IQ3_XXS, 3 GPU -sm tensor, -c 8192, 5256-token prompt upstream pp 1399.7 t/s tg 53.5 t/s text 7a8734a9f1a6 delivery pp 1432.1 t/s tg 70.1 t/s text 7a8734a9f1a6 <-- identical Qwen3.8-27B UD-Q4_K_M, 1 GPU, same prompt upstream pp 1311.1 t/s tg 32.4 t/s text b905c9763a58 delivery pp 1303.2 t/s tg 32.4 t/s text b21e94a5b6e1 (neutral; text diverges, needs the KLD gate to judge) So on the config this repo actually validates the delivery's implementation is BIT-IDENTICAL to upstream's and +31 % decode (+2.3 % prefill); on the 27B it is throughput-neutral and its greedy text diverges at 16 tokens (not a purity claim either way -- the chain differs, and the R17 KLD gate is the oracle). Keeping the delivery's implementation also keeps r235's qwen4exp validation and the two delivery gates (gate-hc-fusions.sh, gate-qwen4exp-quant-coherence.sh) meaningful. Validated: build green, no `<<<<<<<`; dense 4B (1472ebdadd46) and Qwen3-Coder-30B-A3B (013d09f165a5) unchanged; Flash-Next and 27B run to completion on the VMM build; gate-hc-fusions.sh PASS (5/5 hard arms byte-identical to the chain, 449 chars d62340fbf566); gate-qwen4exp-quant-coherence.sh PASS (IQ3_XXS / IQ4_NL / Q4_K_M: gather on == off, no '////'). Known drift to record at R17: on this tree the two REPORTED dsv4_pre/post arms no longer agree with the chain on the BF16-hc class (chain 449 chars d62340fbf566, dsv4 arms 6a102dbc62d6 = the value the chain produced at r235), i.e. the HC chain itself moved with the re-base while the fused pre/post ops still reproduce the old one. The gate reports (does not fail) them, and its header says byte-identity IS expected on this class -- the soft-arm expectation needs a decision.

…vocab N) The MTP draft head scores only the LM-head rows for token ids < N plus every control/user-defined id, then scatters the logits into a full-vocabulary row of -inf: the speculator still sees a full-vocab logit row and the target's verification is untouched, so the trim can only lose proposals, never change an accepted token. Opt-in and default off (-spec-draft-mtp-vocab 0 builds no subset and adds no graph node). Contents: - common: --spec-draft-mtp-vocab N + LLAMA_ARG_SPEC_DRAFT_MTP_VOCAB, plumbed to llama_context_params::mtp_draft_vocab for the MTP draft context only. - src: llama_mtp_draft_vocab (verbatim LM-head row copy in its own quant type + I64 id map), per-context ownership with a model-side weak-ref cache keyed by (N, buffer type), a BPE merge-order guard, and the graph hook in the qwen35 / qwen35moe / qwen4exp / gemma4-assistant MTP graphs. - shared-NextN sidecars (nextn_shared_target_tensors, no output.weight) build the subset from the target's borrowed LM head via cparams.ctx_other. - ggml-backend-meta: an arbitrary ROW RANGE of a 2-D tensor whose rows are split across devices (a vocabulary-split LM head) is spliced block by block. Without it `-sm tensor` aborted in the subset build (a 2-D axis-1 split could only be read whole, because its splice chunk is nb[axis + 1]). - tests/test-mtp-draft-vocab.cpp (built, not ctest-registered: it needs a qwen35/qwen35moe model with an MTP head). Gated on gfx1201 (2x R9700, ROCm 10.1.0), Qwen3.8-27B Q8_0, 2 GPU `-sm tensor`, `draft-mtp-adaptive,ngram-mod --spec-draft-n-max 7 --spec-draft-mtp-vocab 98304`, -n 3000 four-axis: R +1.3 %, P +1.7 %, C +2.4 %, K +0.8 % (+1.6 % aggregate) against the same build with the flag off, with the `n_max <= 7` greedy-purity promise verified with the trim on (three-way text identity vs `--spec-type none` on prose and code). MoE 35B-A3B is neutral. gemma4 is wired up but not beneficial on our payload (its useful cut is 245760 of 262144 rows, a no-op). `test-backend-ops` 21179/21179, `test-logits-width-probe SPLIT=tensor` PASS (W=1..8 maxdiff 0), prefill-logit KLD 0.000000 / 100.000 %. Co-authored-by: Simon Teixidor <simon@flaskpost.me> Co-authored-by: Laurent Zuijdwijk <laurent.zuijdwijk@gmail.com> Co-authored-by: Martino B <martino2k6@users.noreply.github.com> Re-based onto a691b23 (r236). Four files conflicted (the survey predicted three; common/common.cpp is the fourth, an arg-plumbing hunk). * common/common.cpp -- upstream replaced the inline fit block with common_fit_apply(), which already derives the draft's post-slab reserve (the delivery's block-09 fold, upstream absorbed it), so the call site stays upstream's and block 13's one meaningful line -- cparams_dft.mtp_draft_vocab = params.speculative.draft.mtp_vocab -- moves into common_fit_apply()'s existing `if (spec_mtp)`. * src/llama-model.h / src/llama-model.cpp -- both sides add new declarations at the same place: the llama_rope_params getters (base drift) and the new llama_mtp_draft_vocab struct / mtp_draft_vocab_get are kept side by side, and the shared `return res;` tail was needed twice (both conflicted functions end there). * src/llama-graph.h -- llm_graph_params::can_reuse compares both sides' terms. The block's `arch == other.arch` term is dropped: `arch` no longer exists on the params after ggml-org#31733 (the model is a graph-context member and `model == other.model` already implies the arch). * src/llama-context.cpp -- ggml-org#31141 drift: `model.hparams.no_alloc` -> `model.no_alloc()` in the shared-NextN head probe. Exit verified on this tree (gfx1201, ROCm 10.1.0, VMM build): * build green, no conflict markers, coherence unchanged against R13 on four arms -- dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6. * shared-NextN head path (Flash-Next GSQ + mtp-...-shared-Q8_0, 3 GPU -sm tensor): `--spec-draft-mtp-vocab 98304` logs "MTP draft head uses 98337 of 248320 rows (165.06 MiB of q5_K, Meta())" -- i.e. built from the target's borrowed LM head through cparams.ctx_other, and the row-range splice over a vocabulary-split (Meta) tensor that this block adds is exercised -- with identical acceptance (0.49333, 37/75, mean len 2.42) and identical greedy text (55a50dee05e0) to the same run with the flag unset and with `-spec-draft-mtp-vocab 0`. * self-headed path (Gemma4-12B Q8_0 + its own MTP head, 1 GPU): cut 245760 builds 245766 of 262144 rows with acceptance and text identical to off (0.68852, 42/61, mean len 3.00; f8ef9d420320). An aggressive cut (98304, which this tokenizer's distribution does not suit) drops acceptance to 0.31915 (30/94) and leaves the text **identical** -- the promised property: the trim can only lose proposals, never change an accepted token. No prompt-cache or DFlash interaction.