llama : remove obsolete n_pad from memory instances by ggerganov · Pull Request #31200 · ggml-org/llama.cpp

ggerganov · 2026-10-26T14:57:42Z

Overview

n_pad was used to pad the KV cache size to a multiple of 256 for the flash attention backends. This is no longer done since #16812, which kept the constructor argument "although it is not currently used anymore".

Indeed, all callers pass n_pad = 1 and the only 2 users clamp it to a fixed 256:

llama_kv_cache::get_n_kv() - std::max(n_pad, 256u)
llama_kv_cache_msa_context::get_n_pos() - std::max(kv->get_n_pad(), 256u)

Remove the parameter from llama_kv_cache, llama_kv_cache_iswa, llama_kv_cache_dsa, llama_kv_cache_dsa_iswa, llama_kv_cache_dsv4, llama_kv_cache_msa, llama_memory_hybrid, llama_memory_hybrid_iswa and llama_memory_hybrid_idx, together with the members, the llama_kv_cache_msa::get_n_pad() accessor and the trivially true GGML_ASSERT(kv_size % n_pad == 0). The padding of n_kv and n_pos is kept as a constant.

Additional information

No functional change. The removed assert was the one triggered in #14724.

Validated with test-llama-archs and test-llama-context on Metal: 136 dummy models, all passed.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES
- Harness: dsh:llama.cpp
- Hardware: MacBook, M5 Max, 128GB
- Model: ggml-org/Qwen3.8-Flash-Next-GGUF, linked to its Hugging Face page
- Stats: ◷ 10.6m · ⚙ 2.9m · ↑ 55k · ↓ 29k · ⇈ 503.3 tok/s · ⇊ 53.3 tok/s · ≡ 187 · ⌨ 14

Every caller passes n_pad = 1 and the only two users clamp it to 256. Drop the parameter, the members and the trivially true kv_size assert. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)

This reverts commit 74bca39. Upstream's DeepSeek-V4.1 support is replaced by the fork's own implementation (deepseek41, later in this series), which the fork's GGUF files, CUDA kernels and server paths are built for. Upstream changes made after ggml-org#31086 are kept where they are not DeepSeek-V4.1 specific: ggml-org#31142 output structs (build_hc_pre returns hc_pre with the mixes computed inline again), ggml-org#31141 no_alloc and ggml-org#31200 n_pad in the DSV4 cache constructors, ggml-org#31368 rs_max / update_rs_for_ubatches, its tensors and build_dspark_block_mask. tests/llama-context still names LLM_ARCH_DEEPSEEK41, which the fork adds back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…r, QSA/indexer) Re-based onto a691b23: 9 files / 34 hunks (the probe predicted 7 files; the cascades were llama-memory-recurrent.cpp and tests/CMakeLists.txt). Decision D3 was measured, not assumed -- see below; this commit keeps the DELIVERY's implementation. Base-API drift this rung had to re-derive (all compiler-driven, no behaviour change): - llama-kv-cache: the ctor keeps BOTH indexer flags -- upstream's `is_indexer` (indexer RoPE params) and the delivery's `v_enabled` (keys-only caches) -- and drops the delivery's `n_pad` ctor parameter and member: upstream ggml-org#31200 removed them and hardcodes the 256-cell floor in the size math, so only the derived `n_pad_prefill` (LLAMA_KV_N_PAD_PREFILL / _MIN / _FROM) survives. `get_n_kv()` now REQUIRES n_tokens (no silent default: a 0 would turn the prefill pad off at that call site) and llama-kv-cache-dsv4 passes it. - llama-memory-hybrid-idx: the keys-only indexer cache is built with upstream's post-ggml-org#31141 ctor (no_alloc, no n_pad) plus is_indexer/v_enabled; the pool body is the delivery's. - llama-memory-recurrent: R04 wins (upstream's rs_max bound, the tracker stays dropped) and the block's two real additions are kept -- the getenv-once `rs_rollback_debug()` helper (issue ggml-org#273) and the `has_recr` gate that stops the qwen4exp MTP context (whose recurrent filter matches nothing) spamming the non-consecutive-position warning. - models.h / qwen4exp.cpp: ggml-org#31733 made `model` a member of llm_graph_context, so the graph and graph_mtp ctors lose their `const llama_model &` argument (and the class its shadowing `const llama_model & model` member); ggml-org#31073 turned the rotation constants into per-layer vectors (`self_k_rot[il]`) and ggml-org#29510/ggml-org#31683 replaced `get_kq_mask()` with `self_attn_mask.raw`; build_attn_mha grew `n_kv_max`. - indexer-score.cu: `ggml_rope_yarn_corr_dims` gained the truncate flag. - tests/CMakeLists.txt: R03 had already re-placed every registration this block adds (ggml-org#30880 moved the generated-model fixtures), so only `test-lazy-reader` is new; test-llama-archs keeps upstream's get_model/get_model_and_ctx split with the block's `lazy_buf_size` handed over as LLAMA_LAZY_BUF_MB, and test-backend-ops keeps BOTH sides' Flash-Attention classes (upstream's kv_rows and the delivery's QSA) and both perf blocks. D3 (qwen4exp / indexer / sparse selection) -- MEASURED, decision: take the delivery's implementation. Upstream now carries its own qwen4exp (1514 lines, by upstream contributors: ggml-org#30965, ggml-org#31243, ggml-org#31443, ggml-org#30927, ggml-org#31086) with its own HC mix/combine (plain ops), QSA selection (build_qsa_sel, an additive mask) and indexer (plain ops); the delivery's is 3197 lines with fused HC ops (HC_MIX/HC_COMBINE, HC16 planar layout), the QSA store+top-k selection (no mask), INDEXER_SCORE/TOPK/FILL and a keys-only indexer cache. A/B on the same GGUFs, same build: Flash-Next GSQ-IQ3_XXS, 3 GPU -sm tensor, -c 8192, 5256-token prompt upstream pp 1399.7 t/s tg 53.5 t/s text 7a8734a9f1a6 delivery pp 1432.1 t/s tg 70.1 t/s text 7a8734a9f1a6 <-- identical Qwen3.8-27B UD-Q4_K_M, 1 GPU, same prompt upstream pp 1311.1 t/s tg 32.4 t/s text b905c9763a58 delivery pp 1303.2 t/s tg 32.4 t/s text b21e94a5b6e1 (neutral; text diverges, needs the KLD gate to judge) So on the config this repo actually validates the delivery's implementation is BIT-IDENTICAL to upstream's and +31 % decode (+2.3 % prefill); on the 27B it is throughput-neutral and its greedy text diverges at 16 tokens (not a purity claim either way -- the chain differs, and the R17 KLD gate is the oracle). Keeping the delivery's implementation also keeps r235's qwen4exp validation and the two delivery gates (gate-hc-fusions.sh, gate-qwen4exp-quant-coherence.sh) meaningful. Validated: build green, no `<<<<<<<`; dense 4B (1472ebdadd46) and Qwen3-Coder-30B-A3B (013d09f165a5) unchanged; Flash-Next and 27B run to completion on the VMM build; gate-hc-fusions.sh PASS (5/5 hard arms byte-identical to the chain, 449 chars d62340fbf566); gate-qwen4exp-quant-coherence.sh PASS (IQ3_XXS / IQ4_NL / Q4_K_M: gather on == off, no '////'). Known drift to record at R17: on this tree the two REPORTED dsv4_pre/post arms no longer agree with the chain on the BF16-hc class (chain 449 chars d62340fbf566, dsv4 arms 6a102dbc62d6 = the value the chain produced at r235), i.e. the HC chain itself moved with the re-base while the fused pre/post ops still reproduce the old one. The gate reports (does not fail) them, and its header says byte-identity IS expected on this class -- the soft-arm expectation needs a decision.

…W (Goal A) and an attendance POOL (Goal B), both opt-in Re-based onto a691b23 (r236) The rung collided with everything the streamed KV binds to: the memory instances (ggml-org#31141 no_alloc, ggml-org#31200 n_pad removed, ggml-org#31142), the KV cache internals (ggml-org#30541 on-device state copy sizes, ggml-org#31073 the rotation constants moved into a per-cache buffer, ggml-org#31570 kv_rows), the Meta (tensor-split) device (ggml-org#31447) and the qwen4exp rewrite. 8 files conflicted / 12 hunks (9 expected: src/models/qwen4exp.cpp auto-merged, the D3 residue did not conflict this time and its 59/-7 lines all landed). A distinctive-added-line check over the 25 auto-merged files found no dropped delivery code. D13 (memory instances) -- re-derived, not carried: * llama_memory_params keeps upstream's `no_alloc` and adds the delivery's `kv_pool` (llama-context.cpp's designated init passes both); the three `hparams.no_alloc` reads the block added follow ggml-org#31141 out of the hparams and become the ctor's `no_alloc`. * memory_breakdown() is re-derived over `ctxs_bufs.bufs` + `ctxs_bufs.no_alloc` instead of the old per-cache loop: upstream moved the no_alloc/plain accounting into llama_memory_bufs and added attn_rot_bufs (ggml-org#31073), whose bytes are summed on top. The pool's mirror-to-host-bucket and the window's resident/spill split accounting are unchanged. * the MTP draft window's llama_kv_cache construction drops the `n_pad` argument (ggml-org#31200), and the pool's cell count is rounded to whole PAGES (it used to round to the cache's n_pad, which no longer exists as a parameter -- n_ctx is unpadded now). * create_memory() takes upstream's `params` name back (the delivery-side `params_l` rename is gone; the body was upstream's everywhere but two lines). * src/llama-ext.h keeps upstream's classifier-free-guidance declaration, src/llama-memory.h keeps upstream's llama_memory_bufs struct, ggml-backend-meta.cpp keeps upstream's view-slab helpers next to ggml_backend_meta_kv_region_grow. src/llama-context.cpp takes upstream's readback-sync hunk -- the delivery side of it was stale F1 code (layer_inp_host_fill, deleted at R13). D12 (quantized-KV state, rotation) -- the provider still binds at the BUFFER TYPE and no kernel addresses cells differently: the rotation constants are a per-cache, per-layer buffer (attn_rot_bufs), not cell-indexed, so they are outside the region; state_write/state_read keep going through the region buffer's own copy. Proven by the gates below, whose spilling arms really spill (16 mapping flips, 64 MiB host-backed) and whose state arm (win2mtp) takes the region path. The r193 draft reserve is preserved (r236 absorbed the fit plumbing): common/common.cpp takes upstream's common_fit_apply() call, and block 15's mtp_shared branch is folded into common_draft_reserve_derive() -- the one function both `--fit` and the MoE preflight call. Measured, 35B-A3B Q8_0 1 GPU -ncmoe 48 -c 262144 with --spec-type draft-mtp: window auto derives "draft CONTEXT 1102 MiB (kv 142 + compute 768 + slack 192)" -> arena 20527.5 MiB / 62.9 % residency; window 0 keeps the flat 3584 MiB -> arena 17977.5 MiB / 55.1 %. +2550 MiB of arena, +7.8 points. Fixed on the way: * src/llama-memory-hybrid.cpp passed its attention cache to the extended ctor positionally, so after upstream inserted `is_indexer` before `v_enabled` the delivery's `kv_pool` landed in `v_enabled`: the cache allocated NO V tensor and llama_kv_cache::cpy_v null-dereferenced on the first graph build (every hybrid/GDN model segfaulted at load; the dense 4B arm died before generating). `is_indexer` is now named explicitly. The other ctor call sites were audited: hybrid-idx already named it, model.cpp stops at `share`, and dsa/dsa-iswa/msa/dsv4 are upstream's and correct. * ggml/src/ggml-cuda/indexer-score.cu is NOT an R16 drift but a latent R12 defect (b1b0fa807): the fused GGML_OP_INDEXER_SCORE node called ggml_rope_get_truncate(dst), and upstream's ggml-org#30206 added an assert that the argument is a ROPE/ROPE_BACK node. Every qwen4exp sparse QSA decode run has aborted on this tree since R12 (no R12-R15 gate reaches that arm -- the coherence arms and both qwen4exp gates run the dense arm; block 15's residency gate reaches it because its `pref` arm sets LLAMA_QSA_DENSE_DECODE_UNTIL=0, and aborted after 63 generated tokens at ggml.c:4648). Both call sites now pass /*truncate*/ true, which is the graph's own rope node default (ggml_rope_impl initialises no_trunc 0 and nothing calls ggml_rope_set_truncate on the indexer-key node) and the pre-ggml-org#30206 5-argument helper's unconditional floor/ceil, i.e. pre-re-base numerics. Coherence (same session, same commands, this build family): dense 4B 1de99b7e6ee6, 30B-A3B 4b4d56191368, 35B-A3B 3 GPU tensor rccl 86b7f9b5aa80, Flash-Next GSQ 7a8734a9f1a6 -- all four unchanged from R13. Gates: * gate-kv-residency.sh PASS. Window: wref == wres == wspill be904e86f248, wres 0 flips / 0 MiB host-backed, wspill 16 flips / 64 MiB, every slide on the map grid. Pool: pref == pres == pfloor f260d1824ab3, bad 0, 8192 pages (408.0 MiB) on pres vs 4104 (204.4 MiB) on pfloor, 252 resolves both, claims 137.7 -> 217.1 per resolve, hits 91.02 -> 89.63 %. Multi-device (-sm layer 0,1): pmref == pmres == pmfloor f260d1824ab3, one mirror/pool/counter block per device, 2 counter blocks summed, 117.8 -> 268.1 claims per resolve. * gate-kv-window-split.sh PASS, 38/38: one region per device on a 3-device 4B, a 27B at -c 524288 sliding on BOTH devices (32 flips each), -c 1048576 windowed, 1-device unchanged, -nkvo still declines, all five fit arms (fixed/auto/ram/ram-drop/off), and win2mtp == ref2mtp with identical acceptance 0.46154 whose batched state save takes the region path. * gate-spec-draft-window.sh PASS: LLAMA_SPEC_DRAFT_WINDOW=1024 gives "1024 cells per sequence, cache 3072 cells ((win + 512) x 2)" and all six requests across both slots are byte-identical to win 0, acceptance identical (0.47222 / 17-36, 1.0 / 23-23). * check-gpu-faults.sh --since the session start: clean.