llama : add llama_arch_* query API and refactor test-llama-archs to use it by ggerganov · Pull Request #30880 · ggml-org/llama.cpp

ggerganov · 2026-10-20T13:07:42Z

Overview

This PR is a first step towards introducing arch-related API for libllama: the internal llm_ arch query functions get a public llama_arch_* face, declared in the llama-ext.h staging header, so external code can enumerate the known architectures, query their canonical names and capabilities without reaching into internal headers. Follow-up work can extend this surface as needed.

test-llama-archs is refactored as the first consumer of this API: it previously included internal libllama headers (src/llama-arch.h, src/llama-model-saver.h) and referenced symbols that are not exported from the library, which prevented building the test (and the tests consuming its generated models) on Windows with shared libraries. To this end, llama_model_saver is slimmed down to the private implementation of llama_model_save_to_file(), the fixture-writer half that the test was using moves into tests/test-llama-archs.cpp as a local kv_writer, and the roundtrip check switches to the public llama_model_save_to_file() writing to a temp path (also removing the tmpfile() fd leak and its Windows null-case workaround). test-llama-archs, test-generate-models, test-llama-context and test-fusion are moved out of the NOT WIN32 OR NOT BUILD_SHARED_LIBS guard so they are built and run on all platforms.

Additional information

the exported surface grows only by the llama_arch_* query wrappers, llama_arch_kv_name() and llama_arch_supports_save() (the arch/KV enums and LLM_KV remain internal and reach the test through llama-ext.h -> llama-arch.h)
the internal llm_ API is unchanged, the wrappers are thin and behavior-neutral
verified with a shared-lib build (BUILD_SHARED_LIBS=ON): the llama_arch_* symbols are exported, llama_model_saver stays internal, and the test links the shared library cleanly
test-llama-archs smoke runs pass across all backends incl. MoE host-experts and MoE-cache configs, test-generate-models produces all 134 fixture models, test-llama-context passes against them

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES
- Harness: pi:llama.cpp
- Hardware: 2x DGX Spark (RDMA, TP=2)
- Model: ggml-org/GLM-5.3-Flash-GGUF
- Stats: ◷ 1.8h · ⚙ 8.6m · ↑ 97k · ↓ 111k · ≡ 203 · ⌨ 9

test-llama-archs included src/llama-arch.h and src/llama-model-saver.h, which prevented building on Windows with shared libraries since the used symbols are not exported from the DLL. - declare llama_arch_* wrappers around the internal llm_ arch query functions in the llama-ext.h staging header, together with llama_arch_kv_name() and llama_arch_supports_save() - slim llama_model_saver down to the private implementation of llama_model_save_to_file(): the fixture-writer half used by the test moves into tests/test-llama-archs.cpp as a local kv_writer - replace the roundtrip saver usage with the public llama_model_save_to_file() writing to a temp path (also removes the tmpfile() fd leak and its Windows workaround) - move test-llama-archs, test-generate-models, test-llama-context and test-fusion out of the NOT WIN32 OR NOT BUILD_SHARED_LIBS guard Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))

…ls (ggml-org#30880) test-llama-archs included src/llama-arch.h and src/llama-model-saver.h, which prevented building on Windows with shared libraries since the used symbols are not exported from the DLL. - declare llama_arch_* wrappers around the internal llm_ arch query functions in the llama-ext.h staging header, together with llama_arch_kv_name() and llama_arch_supports_save() - slim llama_model_saver down to the private implementation of llama_model_save_to_file(): the fixture-writer half used by the test moves into tests/test-llama-archs.cpp as a local kv_writer - replace the roundtrip saver usage with the public llama_model_save_to_file() writing to a temp path (also removes the tmpfile() fd leak and its Windows workaround) - move test-llama-archs, test-generate-models, test-llama-context and test-fusion out of the NOT WIN32 OR NOT BUILD_SHARED_LIBS guard Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))

Upstream ggml-org#30880 removed the llama_model_saver(arch, gguf_ctx) fixture constructor that this test used to write the GGUF metadata of its tiny model, so the test no longer compiled and broke full builds. Write the keys with a local kv_writer over llama_arch_kv_name(), as test-llama-archs.cpp now does, and use llama_arch_name() from llama-ext.h instead of the internal llm_arch_name().

…r, QSA/indexer) Re-based onto a691b23: 9 files / 34 hunks (the probe predicted 7 files; the cascades were llama-memory-recurrent.cpp and tests/CMakeLists.txt). Decision D3 was measured, not assumed -- see below; this commit keeps the DELIVERY's implementation. Base-API drift this rung had to re-derive (all compiler-driven, no behaviour change): - llama-kv-cache: the ctor keeps BOTH indexer flags -- upstream's `is_indexer` (indexer RoPE params) and the delivery's `v_enabled` (keys-only caches) -- and drops the delivery's `n_pad` ctor parameter and member: upstream ggml-org#31200 removed them and hardcodes the 256-cell floor in the size math, so only the derived `n_pad_prefill` (LLAMA_KV_N_PAD_PREFILL / _MIN / _FROM) survives. `get_n_kv()` now REQUIRES n_tokens (no silent default: a 0 would turn the prefill pad off at that call site) and llama-kv-cache-dsv4 passes it. - llama-memory-hybrid-idx: the keys-only indexer cache is built with upstream's post-ggml-org#31141 ctor (no_alloc, no n_pad) plus is_indexer/v_enabled; the pool body is the delivery's. - llama-memory-recurrent: R04 wins (upstream's rs_max bound, the tracker stays dropped) and the block's two real additions are kept -- the getenv-once `rs_rollback_debug()` helper (issue ggml-org#273) and the `has_recr` gate that stops the qwen4exp MTP context (whose recurrent filter matches nothing) spamming the non-consecutive-position warning. - models.h / qwen4exp.cpp: ggml-org#31733 made `model` a member of llm_graph_context, so the graph and graph_mtp ctors lose their `const llama_model &` argument (and the class its shadowing `const llama_model & model` member); ggml-org#31073 turned the rotation constants into per-layer vectors (`self_k_rot[il]`) and ggml-org#29510/ggml-org#31683 replaced `get_kq_mask()` with `self_attn_mask.raw`; build_attn_mha grew `n_kv_max`. - indexer-score.cu: `ggml_rope_yarn_corr_dims` gained the truncate flag. - tests/CMakeLists.txt: R03 had already re-placed every registration this block adds (ggml-org#30880 moved the generated-model fixtures), so only `test-lazy-reader` is new; test-llama-archs keeps upstream's get_model/get_model_and_ctx split with the block's `lazy_buf_size` handed over as LLAMA_LAZY_BUF_MB, and test-backend-ops keeps BOTH sides' Flash-Attention classes (upstream's kv_rows and the delivery's QSA) and both perf blocks. D3 (qwen4exp / indexer / sparse selection) -- MEASURED, decision: take the delivery's implementation. Upstream now carries its own qwen4exp (1514 lines, by upstream contributors: ggml-org#30965, ggml-org#31243, ggml-org#31443, ggml-org#30927, ggml-org#31086) with its own HC mix/combine (plain ops), QSA selection (build_qsa_sel, an additive mask) and indexer (plain ops); the delivery's is 3197 lines with fused HC ops (HC_MIX/HC_COMBINE, HC16 planar layout), the QSA store+top-k selection (no mask), INDEXER_SCORE/TOPK/FILL and a keys-only indexer cache. A/B on the same GGUFs, same build: Flash-Next GSQ-IQ3_XXS, 3 GPU -sm tensor, -c 8192, 5256-token prompt upstream pp 1399.7 t/s tg 53.5 t/s text 7a8734a9f1a6 delivery pp 1432.1 t/s tg 70.1 t/s text 7a8734a9f1a6 <-- identical Qwen3.8-27B UD-Q4_K_M, 1 GPU, same prompt upstream pp 1311.1 t/s tg 32.4 t/s text b905c9763a58 delivery pp 1303.2 t/s tg 32.4 t/s text b21e94a5b6e1 (neutral; text diverges, needs the KLD gate to judge) So on the config this repo actually validates the delivery's implementation is BIT-IDENTICAL to upstream's and +31 % decode (+2.3 % prefill); on the 27B it is throughput-neutral and its greedy text diverges at 16 tokens (not a purity claim either way -- the chain differs, and the R17 KLD gate is the oracle). Keeping the delivery's implementation also keeps r235's qwen4exp validation and the two delivery gates (gate-hc-fusions.sh, gate-qwen4exp-quant-coherence.sh) meaningful. Validated: build green, no `<<<<<<<`; dense 4B (1472ebdadd46) and Qwen3-Coder-30B-A3B (013d09f165a5) unchanged; Flash-Next and 27B run to completion on the VMM build; gate-hc-fusions.sh PASS (5/5 hard arms byte-identical to the chain, 449 chars d62340fbf566); gate-qwen4exp-quant-coherence.sh PASS (IQ3_XXS / IQ4_NL / Q4_K_M: gather on == off, no '////'). Known drift to record at R17: on this tree the two REPORTED dsv4_pre/post arms no longer agree with the chain on the BF16-hc class (chain 449 chars d62340fbf566, dsv4 arms 6a102dbc62d6 = the value the chain produced at r235), i.e. the HC chain itself moved with the re-base while the fused pre/post ops still reproduce the old one. The gate reports (does not fail) them, and its header says byte-identity IS expected on this class -- the soft-arm expectation needs a decision.

…r, QSA/indexer) Re-based onto a691b23: 9 files / 34 hunks (the probe predicted 7 files; the cascades were llama-memory-recurrent.cpp and tests/CMakeLists.txt). Decision D3 was measured, not assumed -- see below; this commit keeps the DELIVERY's implementation. Base-API drift this rung had to re-derive (all compiler-driven, no behaviour change): - llama-kv-cache: the ctor keeps BOTH indexer flags -- upstream's `is_indexer` (indexer RoPE params) and the delivery's `v_enabled` (keys-only caches) -- and drops the delivery's `n_pad` ctor parameter and member: upstream ggml-org#31200 removed them and hardcodes the 256-cell floor in the size math, so only the derived `n_pad_prefill` (LLAMA_KV_N_PAD_PREFILL / _MIN / _FROM) survives. `get_n_kv()` now REQUIRES n_tokens (no silent default: a 0 would turn the prefill pad off at that call site) and llama-kv-cache-dsv4 passes it. - llama-memory-hybrid-idx: the keys-only indexer cache is built with upstream's post-ggml-org#31141 ctor (no_alloc, no n_pad) plus is_indexer/v_enabled; the pool body is the delivery's. - llama-memory-recurrent: R04 wins (upstream's rs_max bound, the tracker stays dropped) and the block's two real additions are kept -- the getenv-once `rs_rollback_debug()` helper (issue ggml-org#273) and the `has_recr` gate that stops the qwen4exp MTP context (whose recurrent filter matches nothing) spamming the non-consecutive-position warning. - models.h / qwen4exp.cpp: ggml-org#31733 made `model` a member of llm_graph_context, so the graph and graph_mtp ctors lose their `const llama_model &` argument (and the class its shadowing `const llama_model & model` member); ggml-org#31073 turned the rotation constants into per-layer vectors (`self_k_rot[il]`) and ggml-org#29510/ggml-org#31683 replaced `get_kq_mask()` with `self_attn_mask.raw`; build_attn_mha grew `n_kv_max`. - indexer-score.cu: `ggml_rope_yarn_corr_dims` gained the truncate flag. - tests/CMakeLists.txt: R03 had already re-placed every registration this block adds (ggml-org#30880 moved the generated-model fixtures), so only `test-lazy-reader` is new; test-llama-archs keeps upstream's get_model/get_model_and_ctx split with the block's `lazy_buf_size` handed over as LLAMA_LAZY_BUF_MB, and test-backend-ops keeps BOTH sides' Flash-Attention classes (upstream's kv_rows and the delivery's QSA) and both perf blocks. D3 (qwen4exp / indexer / sparse selection) -- MEASURED, decision: take the delivery's implementation. Upstream now carries its own qwen4exp (1514 lines, by upstream contributors: ggml-org#30965, ggml-org#31243, ggml-org#31443, ggml-org#30927, ggml-org#31086) with its own HC mix/combine (plain ops), QSA selection (build_qsa_sel, an additive mask) and indexer (plain ops); the delivery's is 3197 lines with fused HC ops (HC_MIX/HC_COMBINE, HC16 planar layout), the QSA store+top-k selection (no mask), INDEXER_SCORE/TOPK/FILL and a keys-only indexer cache. A/B on the same GGUFs, same build: Flash-Next GSQ-IQ3_XXS, 3 GPU -sm tensor, -c 8192, 5256-token prompt upstream pp 1399.7 t/s tg 53.5 t/s text 7a8734a9f1a6 delivery pp 1432.1 t/s tg 70.1 t/s text 7a8734a9f1a6 <-- identical Qwen3.8-27B UD-Q4_K_M, 1 GPU, same prompt upstream pp 1311.1 t/s tg 32.4 t/s text b905c9763a58 delivery pp 1303.2 t/s tg 32.4 t/s text b21e94a5b6e1 (neutral; text diverges, needs the KLD gate to judge) So on the config this repo actually validates the delivery's implementation is BIT-IDENTICAL to upstream's and +31 % decode (+2.3 % prefill); on the 27B it is throughput-neutral and its greedy text diverges at 16 tokens (not a purity claim either way -- the chain differs, and the R17 KLD gate is the oracle). Keeping the delivery's implementation also keeps r235's qwen4exp validation and the two delivery gates (gate-hc-fusions.sh, gate-qwen4exp-quant-coherence.sh) meaningful. Validated: build green, no `<<<<<<<`; dense 4B (1472ebdadd46) and Qwen3-Coder-30B-A3B (013d09f165a5) unchanged; Flash-Next and 27B run to completion on the VMM build; gate-hc-fusions.sh PASS (5/5 hard arms byte-identical to the chain, 449 chars d62340fbf566); gate-qwen4exp-quant-coherence.sh PASS (IQ3_XXS / IQ4_NL / Q4_K_M: gather on == off, no '////'). Known drift to record at R17: on this tree the two REPORTED dsv4_pre/post arms no longer agree with the chain on the BF16-hc class (chain 449 chars d62340fbf566, dsv4 arms 6a102dbc62d6 = the value the chain produced at r235), i.e. the HC chain itself moved with the re-base while the fused pre/post ops still reproduce the old one. The gate reports (does not fail) them, and its header says byte-identity IS expected on this class -- the soft-arm expectation needs a decision. r242 (contributor PR ggml-org#288, bsp2020): the qwen4exp PREFILL chain is fused and its BF16 GEMM activations are produced, not converted. - `DSV4_HC_POST -> RMS_NORM -> MUL(gamma)` (only emitted under a Meta device, `fused_dsv4_hc_post`) becomes one launch, `dsv4_hc_post_rms_f32` in norm.cu: the hc_post sum with `dsv4_hc_post_f32`'s own two statements, stored (it is the next layer's residual), then `rms_norm_f32`'s SCALAR path (1024-thread, tid-strided) reduction, scale and gamma product -- bit-identical, not rounding-level. Matcher in ggml-cuda-try-fuse.cu (its range check uses the tensor span and allows only the hc_post/residual in-place pair; the window is not contiguous, so the shared check does not apply). ggml-cuda.cu's graph optimizer keeps x/residual/post alive until the MUL. `GGML_CUDA_HC_POST_RMS=0`; census `GGML_CUDA_HC_POST_RMS_DEBUG=1` (INFO: 1302 fused=1, 0 declined on a 31.5k prefill -- a gate on a silent fusion is vacuous, INVARIANTS r225/r238). - `dsv4_hc_post_rms`'s xn and `dsv4_hc_pre`'s mixed stream write the RNE BF16 copy of their output in the same pass and leave it in `bf16_src1_cache`, so the rocBLAS BF16 hc GEMMs skip their F32 -> BF16 conversion. `GGML_CUDA_BF16_PRODUCER=0` off, `=2` re-converts every hit and compares (diagnostic only). Validated (2 R9700, `-sm tensor`, Flash-Next GSQ-RCO IQ3_S): prefill 2330 -> 2470 t/s (+6.0 %, 31.5k prompt), 2421 -> 2617 (+8.1 %, 101k), +6.8 % on the delivery-style arm; greedy text identical in every arm and on the MTP decode arm; prefill-logit KLD -0.000000 / 100.000 %; gate-hc-fusions.sh PASS; HC op oracles 30/30; gfx1100 and gfx1151 text identical with the fusion firing (372 fused, 0 declined); 1 device neutral. Record: archive/work/hc-post-rms-bf16/, WORKLOG 261109.007, INVARIANTS r242.

…r, QSA/indexer) Re-based onto a691b23: 9 files / 34 hunks (the probe predicted 7 files; the cascades were llama-memory-recurrent.cpp and tests/CMakeLists.txt). Decision D3 was measured, not assumed -- see below; this commit keeps the DELIVERY's implementation. Base-API drift this rung had to re-derive (all compiler-driven, no behaviour change): - llama-kv-cache: the ctor keeps BOTH indexer flags -- upstream's `is_indexer` (indexer RoPE params) and the delivery's `v_enabled` (keys-only caches) -- and drops the delivery's `n_pad` ctor parameter and member: upstream ggml-org#31200 removed them and hardcodes the 256-cell floor in the size math, so only the derived `n_pad_prefill` (LLAMA_KV_N_PAD_PREFILL / _MIN / _FROM) survives. `get_n_kv()` now REQUIRES n_tokens (no silent default: a 0 would turn the prefill pad off at that call site) and llama-kv-cache-dsv4 passes it. - llama-memory-hybrid-idx: the keys-only indexer cache is built with upstream's post-ggml-org#31141 ctor (no_alloc, no n_pad) plus is_indexer/v_enabled; the pool body is the delivery's. - llama-memory-recurrent: R04 wins (upstream's rs_max bound, the tracker stays dropped) and the block's two real additions are kept -- the getenv-once `rs_rollback_debug()` helper (issue ggml-org#273) and the `has_recr` gate that stops the qwen4exp MTP context (whose recurrent filter matches nothing) spamming the non-consecutive-position warning. - models.h / qwen4exp.cpp: ggml-org#31733 made `model` a member of llm_graph_context, so the graph and graph_mtp ctors lose their `const llama_model &` argument (and the class its shadowing `const llama_model & model` member); ggml-org#31073 turned the rotation constants into per-layer vectors (`self_k_rot[il]`) and ggml-org#29510/ggml-org#31683 replaced `get_kq_mask()` with `self_attn_mask.raw`; build_attn_mha grew `n_kv_max`. - indexer-score.cu: `ggml_rope_yarn_corr_dims` gained the truncate flag. - tests/CMakeLists.txt: R03 had already re-placed every registration this block adds (ggml-org#30880 moved the generated-model fixtures), so only `test-lazy-reader` is new; test-llama-archs keeps upstream's get_model/get_model_and_ctx split with the block's `lazy_buf_size` handed over as LLAMA_LAZY_BUF_MB, and test-backend-ops keeps BOTH sides' Flash-Attention classes (upstream's kv_rows and the delivery's QSA) and both perf blocks. D3 (qwen4exp / indexer / sparse selection) -- MEASURED, decision: take the delivery's implementation. Upstream now carries its own qwen4exp (1514 lines, by upstream contributors: ggml-org#30965, ggml-org#31243, ggml-org#31443, ggml-org#30927, ggml-org#31086) with its own HC mix/combine (plain ops), QSA selection (build_qsa_sel, an additive mask) and indexer (plain ops); the delivery's is 3197 lines with fused HC ops (HC_MIX/HC_COMBINE, HC16 planar layout), the QSA store+top-k selection (no mask), INDEXER_SCORE/TOPK/FILL and a keys-only indexer cache. A/B on the same GGUFs, same build: Flash-Next GSQ-IQ3_XXS, 3 GPU -sm tensor, -c 8192, 5256-token prompt upstream pp 1399.7 t/s tg 53.5 t/s text 7a8734a9f1a6 delivery pp 1432.1 t/s tg 70.1 t/s text 7a8734a9f1a6 <-- identical Qwen3.8-27B UD-Q4_K_M, 1 GPU, same prompt upstream pp 1311.1 t/s tg 32.4 t/s text b905c9763a58 delivery pp 1303.2 t/s tg 32.4 t/s text b21e94a5b6e1 (neutral; text diverges, needs the KLD gate to judge) So on the config this repo actually validates the delivery's implementation is BIT-IDENTICAL to upstream's and +31 % decode (+2.3 % prefill); on the 27B it is throughput-neutral and its greedy text diverges at 16 tokens (not a purity claim either way -- the chain differs, and the R17 KLD gate is the oracle). Keeping the delivery's implementation also keeps r235's qwen4exp validation and the two delivery gates (gate-hc-fusions.sh, gate-qwen4exp-quant-coherence.sh) meaningful. Validated: build green, no `<<<<<<<`; dense 4B (1472ebdadd46) and Qwen3-Coder-30B-A3B (013d09f165a5) unchanged; Flash-Next and 27B run to completion on the VMM build; gate-hc-fusions.sh PASS (5/5 hard arms byte-identical to the chain, 449 chars d62340fbf566); gate-qwen4exp-quant-coherence.sh PASS (IQ3_XXS / IQ4_NL / Q4_K_M: gather on == off, no '////'). Known drift to record at R17: on this tree the two REPORTED dsv4_pre/post arms no longer agree with the chain on the BF16-hc class (chain 449 chars d62340fbf566, dsv4 arms 6a102dbc62d6 = the value the chain produced at r235), i.e. the HC chain itself moved with the re-base while the fused pre/post ops still reproduce the old one. The gate reports (does not fail) them, and its header says byte-identity IS expected on this class -- the soft-arm expectation needs a decision. r242 (contributor PR ggml-org#288, bsp2020): the qwen4exp PREFILL chain is fused and its BF16 GEMM activations are produced, not converted. - `DSV4_HC_POST -> RMS_NORM -> MUL(gamma)` (only emitted under a Meta device, `fused_dsv4_hc_post`) becomes one launch, `dsv4_hc_post_rms_f32` in norm.cu: the hc_post sum with `dsv4_hc_post_f32`'s own two statements, stored (it is the next layer's residual), then `rms_norm_f32`'s SCALAR path (1024-thread, tid-strided) reduction, scale and gamma product -- bit-identical, not rounding-level. Matcher in ggml-cuda-try-fuse.cu (its range check uses the tensor span and allows only the hc_post/residual in-place pair; the window is not contiguous, so the shared check does not apply). ggml-cuda.cu's graph optimizer keeps x/residual/post alive until the MUL. `GGML_CUDA_HC_POST_RMS=0`; census `GGML_CUDA_HC_POST_RMS_DEBUG=1` (INFO: 1302 fused=1, 0 declined on a 31.5k prefill -- a gate on a silent fusion is vacuous, INVARIANTS r225/r238). - `dsv4_hc_post_rms`'s xn and `dsv4_hc_pre`'s mixed stream write the RNE BF16 copy of their output in the same pass and leave it in `bf16_src1_cache`, so the rocBLAS BF16 hc GEMMs skip their F32 -> BF16 conversion. `GGML_CUDA_BF16_PRODUCER=0` off, `=2` re-converts every hit and compares (diagnostic only). Validated (2 R9700, `-sm tensor`, Flash-Next GSQ-RCO IQ3_S): prefill 2330 -> 2470 t/s (+6.0 %, 31.5k prompt), 2421 -> 2617 (+8.1 %, 101k), +6.8 % on the delivery-style arm; greedy text identical in every arm and on the MTP decode arm; prefill-logit KLD -0.000000 / 100.000 %; gate-hc-fusions.sh PASS; HC op oracles 30/30; gfx1100 and gfx1151 text identical with the fusion firing (372 fused, 0 declined); 1 device neutral. Record: archive/work/hc-post-rms-bf16/, WORKLOG 261109.007, INVARIANTS r242. r243 (contributor PR ggml-org#289, bsp2020): the host PLE gather fans out over --threads and writes the graph input in place. - `llm_graph_input_ple::set_input`: the rows of `per_layer_token_embd` are dequantized straight into `emb->data` when the input tensor lives in a host buffer, so the whole-input host memcpy per prompt ubatch is gone (the buffer is resolved the way `ggml_backend_tensor_set()` resolves it, view_src first, so a view cannot be mis-detected; a device, split or RPC buffer keeps the scratch + `ggml_backend_tensor_set` path). `LLAMA_PLE_DIRECT=0` restores it. - The gather is split over one thread per 4096 rows, so fewer than 8192 rows (decode) stays on the calling thread; the width is the smallest of 8, `std::thread::hardware_concurrency()` and `cparams.n_threads_batch` (i.e. `--threads`/`-tb`, which is what decides it), with `LLAMA_PLE_THREADS` overriding the ceiling for the A/B. Each row is dequantized on its own with the same `to_float`, so the bytes are identical; a worker that cannot be started leaves its rows to the caller (the ranges cover every row whether or not all of them start). ggml's threadpool only ever executes a `ggml_graph_compute`, and it is idle here (the GPUs run the prefill), so the fan-out spawns its own threads exactly as `llama_lazy_reader::gather()` and `llama_state_copy_jobs` already do. Census `LLAMA_PLE_DEBUG=1` (INFO: `ple gather: <rows> rows, <threads> threads, direct=1`), because a silent optimisation makes a later arm vacuous. - Measured (2x R9700 `0,2`, `-sm tensor`, Flash-Next GSQ-RCO IQ3_S, 31.5k prompt, `-ncmoe 47 -lme 47`, `--load-mode mmap`): the delivery shape (`-ub 4096` + attendance pool) **2338.5 -> 2369.2 t/s (+1.31 %)**, 6 runs per side with no overlap between the arms (and +0.77 % again with the 7-thread ceiling `-t 7` sets); `-ub 4096` without the pool 2286.8 -> 2313.3 (+1.16 %); the contributor's own `-ub 6144` field shape 2384.3 -> 2385.7 (+0.06 %) and its 101k prompt 2444.1 -> 2451.9 (+0.32 %) -- the win is `-ub 4096`-shaped (more ubatch boundaries), not a field-shape one. Isolated on one binary with both switches off as the control: threads alone +0.69 %, direct write alone +0.17 %. `llama-batched-bench -npl 1,4,8 -npp 512 -ntg 32`: PP 592.5 -> 597.2 / 2226.0 -> 2242.9 / 2465.0 -> 2539.8 t/s, decode 56.0/160.3/237.1 -> 56.2/157.5/240.7 (inside the spread). - Purity: greedy text identical in every arm (31.5k `9de3ae5502bc`; `draft-mtp n3` at -n 256 `e6ecbe2cd1c8`; 1 device `-sm layer` `0f5005a4398f`; a Qwen3.5-4B control `83eec5e9b4f0`); prefill-logit KLD **-0.000000 / 100.000 %** (`--sm tensor --device 0,2`, base recorded from the r242 build with the same arm) -- the logits are unchanged, not merely inside tolerance; no GPU faults over every run (`check-gpu-faults.sh`). - The contributor's motivation overstated the work per ubatch (~300k rows and a 200 MB copy at `-ub 4096`; `ple_n_heads` is 16, so it is 65 536 rows / 41.9 MB, and 98 304 / 62.9 MB at `-ub 6144`), and their field-shape deltas do not reproduce here; their `-ub 4096` numbers do, and that is the delivery's own shape. Record: archive/work/ple-gather/, WORKLOG 261109.008, INVARIANTS r243, ENVIRONMENT.md.

ggerganov force-pushed the gg/test-llama-archs-public-api branch from 5f9a33f to 7174e02 Compare October 20, 2026 14:07

github-actions Bot added the testing Everything test related label Oct 20, 2026

ravel7524 mentioned this pull request Oct 20, 2026

model-saver: fix per-layer array lengths for MTP models #30891

Draft

ggerganov force-pushed the gg/test-llama-archs-public-api branch from 7174e02 to 10b8b25 Compare October 24, 2026 10:11

ggerganov commented Oct 24, 2026

View reviewed changes

Comment thread src/llama-ext.h Resolved

Comment thread src/llama-ext.h Resolved

ggerganov force-pushed the gg/test-llama-archs-public-api branch from 10b8b25 to cb96d22 Compare October 24, 2026 10:16

ggerganov marked this pull request as ready for review October 24, 2026 10:17

ggerganov requested a review from CISC as a code owner October 24, 2026 10:17

ggerganov force-pushed the gg/test-llama-archs-public-api branch from cb96d22 to 81c007e Compare October 25, 2026 12:33

ggerganov force-pushed the gg/test-llama-archs-public-api branch from 81c007e to eaba687 Compare October 25, 2026 12:55

ggerganov merged commit fd8f845 into master Oct 26, 2026
31 of 34 checks passed

ggerganov deleted the gg/test-llama-archs-public-api branch October 26, 2026 10:16

This was referenced Oct 27, 2026

model : DeepSeek-V4.1-Flash-1017 pin on upstream deepseek41 (MXFP8 Engram + extras) unslothai/llama.cpp#332

Open

pr-set : swap the DeepSeek-V4.1 pin #298 for #332 unslothai/llama.cpp#333

Merged

danielhanchen mentioned this pull request Oct 28, 2026

pr-set : repin DiffusionGemma, Inkling, #241 and #251 for upstream b11946 unslothai/llama.cpp#334

Merged

benjigill mentioned this pull request Oct 28, 2026

Sync upstream master (2026-10-28, ggml-org ac8e1694d) benjigill/llama.cpp#27

Merged

6 tasks

Navigation Menu

Repository navigation

llama : add llama_arch_* query API and refactor test-llama-archs to use it - #30880

llama : add llama_arch_* query API and refactor test-llama-archs to use it#30880
ggerganov merged 1 commit into
masterfrom
gg/test-llama-archs-public-api

ggerganov commented Oct 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ggerganov commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ggerganov commented Oct 20, 2026 •

edited

Loading