llama : move no_alloc out of llama_hparams by ggerganov · Pull Request #31141 · ggml-org/llama.cpp

ggerganov · 2026-10-25T10:37:08Z

Overview

no_alloc is a load/context option, not a model hyperparameter. Remove llama_hparams::no_alloc and pass it explicitly instead:

memory module constructors take a bool no_alloc parameter right after the model reference; wrappers forward it to their sub-caches
llama_memory_params gains a no_alloc field so create_memory threads it without reading it from the model
add llama_memory_bufs (llama-memory.h/.cpp): owns the {ggml context, backend buffer} pairs of a memory instance and centralizes what every leaf previously duplicated - buffer allocation, the no_alloc dummy-buffer path (size-0 buffer with the tensors attached so the scheduler won't allocate them), buffer clear, and memory_breakdown() (real buffer sizes, or simulated sizes via ggml_backend_alloc_ctx_tensors_from_buft_size when no_alloc). Used by llama_kv_cache, llama_memory_recurrent and llama_dsv4_comp_state
llama_moe_cache takes no_alloc as a parameter and reuses llama_memory_bufs::alloc_buf for its own buffer layout
llama_memory_get_buffers() returns an empty vector for a null memory (memory-less archs)

Resolves the TODO added in #31087.

Additional information

test-llama-archs: 538 passed
test-llama-context over all 135 dummy models generated by test-llama-archs -o: 135/135 passed
a no_alloc=true sweep loading every dummy model with LLAMA_LOAD_MODE_NONE: all report a non-zero simulated memory breakdown and only null-base buffers

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES
- Harness: dsh:llama.cpp
- Hardware: 2x DGX Spark (RDMA, TP=2)
- Model: ggml-org/GLM-5.3-Flash-GGUF
- Stats: ◷ 2.7h · ⚙ 6.7m · ↑ 314k · ↓ 139k · ⇈ 123.9 tok/s · ⇊ 17.8 tok/s · ≡ 688 · ⌨ 43

Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))

Upstream moved no_alloc off llama_hparams in ggml-org#31141 and threaded it through the llama_moe_cache constructor instead. The merge updated the signature and the callers, but the host expert pinning added on this branch still read model.hparams.no_alloc, so the file stopped compiling as soon as it was touched. The argument is already in scope right there and the buffer allocation a few lines above uses it, so use it. Assisted-by: Claude Opus 5

* llama : move no_alloc out of llama_hparams Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2)) * cont : clean-up * cont : clean-up Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2)) * tests : add a no_alloc case to test-llama-context Assisted-by: dsh:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))

…, realtime Conflicts resolved by hand: - ggml-cpu.c: keep fork's i02 < 0 skip (MoE hcache uses negative ids) - vecdotq.cuh: keep fork's PrismML 128-block Q1_0/Q2_0 vec_dot - server.cpp: take upstream entrypoint refactor, re-add fork /grammar routes - tests/CMakeLists.txt: take upstream test layout, keep test-kv-mean-center - llama-context.cpp: hcache gate uses model.no_alloc() (ggml-org#31141)

This reverts commit 74bca39. Upstream's DeepSeek-V4.1 support is replaced by the fork's own implementation (deepseek41, later in this series), which the fork's GGUF files, CUDA kernels and server paths are built for. Upstream changes made after ggml-org#31086 are kept where they are not DeepSeek-V4.1 specific: ggml-org#31142 output structs (build_hc_pre returns hc_pre with the mixes computed inline again), ggml-org#31141 no_alloc and ggml-org#31200 n_pad in the DSV4 cache constructors, ggml-org#31368 rs_max / update_rs_for_ubatches, its tensors and build_dspark_block_mask. tests/llama-context still names LLM_ARCH_DEEPSEEK41, which the fork adds back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…b and cache engine) Re-based onto a691b23: 7 conflicted files / 12 hunks, all of them upstream's new subsystem-meets-ours rather than a mechanism fight. - common/common.cpp, common/fit.cpp: upstream factored the --fit block into common_fit_apply(); the delivery's draft reserve (wip/slab-transient-absorption) moves INTO it as a `common_draft_reserve_derive()` helper shared with the ctor's MoE preflight, and fit.cpp keeps both sides (upstream's RAII common_fit_log_scope + the delivery's host-expert accounting and its `hp_ngl_skip` / host-expert out-params, which also fixes the call sites that predated upstream's n_skip parameter). - ggml-backend.cpp: upstream's prefetch-split early-out and the delivery's graph-input sync condition coexist; both event families are freed at teardown. - ggml-cuda/common.cuh: the delivery's keyed CUDA-graph cache with upstream's idle eviction window (ggml-org#30662), keyed comparison instead of the node pointer. - ggml-backend-meta.cpp: the delivery's GGML_META_COMPUTE_HEADROOM knob plus upstream's 256-view floor (ggml-org#31447). - llama-context.cpp / llama-cparams.h: upstream's kv_rows auto-resolution and moe_cache_cpu_share, the delivery's drop_compute_buffers, fused_hc_mix/combine and host-table accounting; `model.hparams.no_alloc` -> `model.no_alloc()` (ggml-org#31141). Finding (blocks the host-expert tier on a default build): the movable-boundary slab and therefore the whole expert-cache arena require VMM, and upstream's HIP default is GGML_HIP_NO_VMM=ON. On build-rocm the arena declines ("the device does not report VMM") and the run streams the experts, so any host-expert measurement needs -DGGML_HIP_NO_VMM=OFF. A second build dir (build-rocm-vmm) was made for the D1 numbers. Validated: build green; dense coherence green; and on Qwen3.6-35B-A3B Q8_0 (-ncmoe 20, -c 8192, 16.3 GiB host experts, 1 GPU, VMM build) at equal MiB the arena (4080 MiB) gives prefill 547.6 / decode 53.5 t/s against upstream --moe-cache-mib 4096's 544.2 / 62.9 and streaming's 470 / 49, with byte-identical text (533 chars, sha 8d6e74e98d89) in every arm. That is D1's first measurement, not its decision.

…r, QSA/indexer) Re-based onto a691b23: 9 files / 34 hunks (the probe predicted 7 files; the cascades were llama-memory-recurrent.cpp and tests/CMakeLists.txt). Decision D3 was measured, not assumed -- see below; this commit keeps the DELIVERY's implementation. Base-API drift this rung had to re-derive (all compiler-driven, no behaviour change): - llama-kv-cache: the ctor keeps BOTH indexer flags -- upstream's `is_indexer` (indexer RoPE params) and the delivery's `v_enabled` (keys-only caches) -- and drops the delivery's `n_pad` ctor parameter and member: upstream ggml-org#31200 removed them and hardcodes the 256-cell floor in the size math, so only the derived `n_pad_prefill` (LLAMA_KV_N_PAD_PREFILL / _MIN / _FROM) survives. `get_n_kv()` now REQUIRES n_tokens (no silent default: a 0 would turn the prefill pad off at that call site) and llama-kv-cache-dsv4 passes it. - llama-memory-hybrid-idx: the keys-only indexer cache is built with upstream's post-ggml-org#31141 ctor (no_alloc, no n_pad) plus is_indexer/v_enabled; the pool body is the delivery's. - llama-memory-recurrent: R04 wins (upstream's rs_max bound, the tracker stays dropped) and the block's two real additions are kept -- the getenv-once `rs_rollback_debug()` helper (issue ggml-org#273) and the `has_recr` gate that stops the qwen4exp MTP context (whose recurrent filter matches nothing) spamming the non-consecutive-position warning. - models.h / qwen4exp.cpp: ggml-org#31733 made `model` a member of llm_graph_context, so the graph and graph_mtp ctors lose their `const llama_model &` argument (and the class its shadowing `const llama_model & model` member); ggml-org#31073 turned the rotation constants into per-layer vectors (`self_k_rot[il]`) and ggml-org#29510/ggml-org#31683 replaced `get_kq_mask()` with `self_attn_mask.raw`; build_attn_mha grew `n_kv_max`. - indexer-score.cu: `ggml_rope_yarn_corr_dims` gained the truncate flag. - tests/CMakeLists.txt: R03 had already re-placed every registration this block adds (ggml-org#30880 moved the generated-model fixtures), so only `test-lazy-reader` is new; test-llama-archs keeps upstream's get_model/get_model_and_ctx split with the block's `lazy_buf_size` handed over as LLAMA_LAZY_BUF_MB, and test-backend-ops keeps BOTH sides' Flash-Attention classes (upstream's kv_rows and the delivery's QSA) and both perf blocks. D3 (qwen4exp / indexer / sparse selection) -- MEASURED, decision: take the delivery's implementation. Upstream now carries its own qwen4exp (1514 lines, by upstream contributors: ggml-org#30965, ggml-org#31243, ggml-org#31443, ggml-org#30927, ggml-org#31086) with its own HC mix/combine (plain ops), QSA selection (build_qsa_sel, an additive mask) and indexer (plain ops); the delivery's is 3197 lines with fused HC ops (HC_MIX/HC_COMBINE, HC16 planar layout), the QSA store+top-k selection (no mask), INDEXER_SCORE/TOPK/FILL and a keys-only indexer cache. A/B on the same GGUFs, same build: Flash-Next GSQ-IQ3_XXS, 3 GPU -sm tensor, -c 8192, 5256-token prompt upstream pp 1399.7 t/s tg 53.5 t/s text 7a8734a9f1a6 delivery pp 1432.1 t/s tg 70.1 t/s text 7a8734a9f1a6 <-- identical Qwen3.8-27B UD-Q4_K_M, 1 GPU, same prompt upstream pp 1311.1 t/s tg 32.4 t/s text b905c9763a58 delivery pp 1303.2 t/s tg 32.4 t/s text b21e94a5b6e1 (neutral; text diverges, needs the KLD gate to judge) So on the config this repo actually validates the delivery's implementation is BIT-IDENTICAL to upstream's and +31 % decode (+2.3 % prefill); on the 27B it is throughput-neutral and its greedy text diverges at 16 tokens (not a purity claim either way -- the chain differs, and the R17 KLD gate is the oracle). Keeping the delivery's implementation also keeps r235's qwen4exp validation and the two delivery gates (gate-hc-fusions.sh, gate-qwen4exp-quant-coherence.sh) meaningful. Validated: build green, no `<<<<<<<`; dense 4B (1472ebdadd46) and Qwen3-Coder-30B-A3B (013d09f165a5) unchanged; Flash-Next and 27B run to completion on the VMM build; gate-hc-fusions.sh PASS (5/5 hard arms byte-identical to the chain, 449 chars d62340fbf566); gate-qwen4exp-quant-coherence.sh PASS (IQ3_XXS / IQ4_NL / Q4_K_M: gather on == off, no '////'). Known drift to record at R17: on this tree the two REPORTED dsv4_pre/post arms no longer agree with the chain on the BF16-hc class (chain 449 chars d62340fbf566, dsv4 arms 6a102dbc62d6 = the value the chain produced at r235), i.e. the HC chain itself moved with the re-base while the fused pre/post ops still reproduce the old one. The gate reports (does not fail) them, and its header says byte-identity IS expected on this class -- the soft-arm expectation needs a decision.

…vocab N) The MTP draft head scores only the LM-head rows for token ids < N plus every control/user-defined id, then scatters the logits into a full-vocabulary row of -inf: the speculator still sees a full-vocab logit row and the target's verification is untouched, so the trim can only lose proposals, never change an accepted token. Opt-in and default off (-spec-draft-mtp-vocab 0 builds no subset and adds no graph node). Contents: - common: --spec-draft-mtp-vocab N + LLAMA_ARG_SPEC_DRAFT_MTP_VOCAB, plumbed to llama_context_params::mtp_draft_vocab for the MTP draft context only. - src: llama_mtp_draft_vocab (verbatim LM-head row copy in its own quant type + I64 id map), per-context ownership with a model-side weak-ref cache keyed by (N, buffer type), a BPE merge-order guard, and the graph hook in the qwen35 / qwen35moe / qwen4exp / gemma4-assistant MTP graphs. - shared-NextN sidecars (nextn_shared_target_tensors, no output.weight) build the subset from the target's borrowed LM head via cparams.ctx_other. - ggml-backend-meta: an arbitrary ROW RANGE of a 2-D tensor whose rows are split across devices (a vocabulary-split LM head) is spliced block by block. Without it `-sm tensor` aborted in the subset build (a 2-D axis-1 split could only be read whole, because its splice chunk is nb[axis + 1]). - tests/test-mtp-draft-vocab.cpp (built, not ctest-registered: it needs a qwen35/qwen35moe model with an MTP head). Gated on gfx1201 (2x R9700, ROCm 10.1.0), Qwen3.8-27B Q8_0, 2 GPU `-sm tensor`, `draft-mtp-adaptive,ngram-mod --spec-draft-n-max 7 --spec-draft-mtp-vocab 98304`, -n 3000 four-axis: R +1.3 %, P +1.7 %, C +2.4 %, K +0.8 % (+1.6 % aggregate) against the same build with the flag off, with the `n_max <= 7` greedy-purity promise verified with the trim on (three-way text identity vs `--spec-type none` on prose and code). MoE 35B-A3B is neutral. gemma4 is wired up but not beneficial on our payload (its useful cut is 245760 of 262144 rows, a no-op). `test-backend-ops` 21179/21179, `test-logits-width-probe SPLIT=tensor` PASS (W=1..8 maxdiff 0), prefill-logit KLD 0.000000 / 100.000 %. Co-authored-by: Simon Teixidor <simon@flaskpost.me> Co-authored-by: Laurent Zuijdwijk <laurent.zuijdwijk@gmail.com> Co-authored-by: Martino B <martino2k6@users.noreply.github.com> Re-based onto a691b23 (r236). Four files conflicted (the survey predicted three; common/common.cpp is the fourth, an arg-plumbing hunk). * common/common.cpp -- upstream replaced the inline fit block with common_fit_apply(), which already derives the draft's post-slab reserve (the delivery's block-09 fold, upstream absorbed it), so the call site stays upstream's and block 13's one meaningful line -- cparams_dft.mtp_draft_vocab = params.speculative.draft.mtp_vocab -- moves into common_fit_apply()'s existing `if (spec_mtp)`. * src/llama-model.h / src/llama-model.cpp -- both sides add new declarations at the same place: the llama_rope_params getters (base drift) and the new llama_mtp_draft_vocab struct / mtp_draft_vocab_get are kept side by side, and the shared `return res;` tail was needed twice (both conflicted functions end there). * src/llama-graph.h -- llm_graph_params::can_reuse compares both sides' terms. The block's `arch == other.arch` term is dropped: `arch` no longer exists on the params after ggml-org#31733 (the model is a graph-context member and `model == other.model` already implies the arch). * src/llama-context.cpp -- ggml-org#31141 drift: `model.hparams.no_alloc` -> `model.no_alloc()` in the shared-NextN head probe. Exit verified on this tree (gfx1201, ROCm 10.1.0, VMM build): * build green, no conflict markers, coherence unchanged against R13 on four arms -- dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6. * shared-NextN head path (Flash-Next GSQ + mtp-...-shared-Q8_0, 3 GPU -sm tensor): `--spec-draft-mtp-vocab 98304` logs "MTP draft head uses 98337 of 248320 rows (165.06 MiB of q5_K, Meta())" -- i.e. built from the target's borrowed LM head through cparams.ctx_other, and the row-range splice over a vocabulary-split (Meta) tensor that this block adds is exercised -- with identical acceptance (0.49333, 37/75, mean len 2.42) and identical greedy text (55a50dee05e0) to the same run with the flag unset and with `-spec-draft-mtp-vocab 0`. * self-headed path (Gemma4-12B Q8_0 + its own MTP head, 1 GPU): cut 245760 builds 245766 of 262144 rows with acceptance and text identical to off (0.68852, 42/61, mean len 3.00; f8ef9d420320). An aggressive cut (98304, which this tokenizer's distribution does not suit) drops acceptance to 0.31915 (30/94) and leaves the text **identical** -- the promised property: the trim can only lose proposals, never change an accepted token. No prompt-cache or DFlash interaction.

…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.