model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support by YanissAmz · Pull Request #25444 · ggml-org/llama.cpp

YanissAmz · 2026-07-08T13:55:53Z

Note: I'm unable to post comments on this repository, so review feedback is answered by pushing the change rather than by replying in the thread.

Overview

Adds support for NVIDIA's Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzleForCausalLM), a hybrid mamba2/attention/MoE model with a per-layer heterogeneous MoE configuration.

Puzzle reuses the existing nemotron_h_moe machinery (latent MoE, relu2, routed scaling, hybrid cache) but is heterogeneous per layer: 88 blocks = 40 mamba2 + 8 attention (GQA 32/2, NoPE) + 40 MoE where moe_intermediate_size (1280–2688) and num_experts_per_tok (4–18) vary per block.

Scope note: this PR was slimmed down on request — the MTP/NEXTN draft-head support it originally carried has been removed and will come back as a separate follow-up PR once this lands, so the two can be reviewed independently. The removal covers the converter as well, not just the model code: it was emitting the nextn tensors, which would have produced GGUFs this loader rejects. mtp.* is now skipped for Puzzle. Note this is no longer the same as "what the other NemotronH variants do": #26725 landed MTP export on NemotronHModel after this branch was written, so the base class now keeps the head. Puzzle stays opted out explicitly (supports_mtp_export = False), since its head is laid out by mtp_block_configs rather than the mtp.layers.* form the base maps, and there is no Puzzle MTP inference path in tree.

Changes:

Per-layer n_ff_exp / n_expert_used — the existing GGUF keys become scalar-OR-array via get_key_or_arr (same pattern as n_ff_arr). Scalar GGUFs broadcast, so this is 100% backward compatible and adds no new keys. build_moe_ffn aggregation is bounded by the per-layer top-k.
Mechanical rename — n_ff_exp → n_ff_exp_impl, n_expert_used → n_expert_used_impl, in its own isolated commit. This is what makes most of the file count: the scalar accessors had to give up their names to the per-layer ones, so every arch touching them is updated in a single no-behaviour-change commit.
Converter — NemotronHPuzzleModel parses block_configs, writes the per-layer arrays, dequantizes the FP8 checkpoint (weight_scale absorption), and accepts both tensor namings in the wild: the original backbone.* prefix and the officially distributed BF16 upload's model.* (model.layers.*, model.embeddings, model.norm_f) with e_score_correction_bias instead of e_score_correction.bias. Also fixes a pre-existing _experts AttributeError on NemotronHModel.
75B.A9B vs 120B.A12B — both have 88 layers; a heterogeneity check on the per-layer top-k array distinguishes them. Super is uniform, so it still resolves to 120B.A12B and keeps working as before.

Additional information

Validation. The full-scope version of this branch was validated end to end on Strix Halo 128GB unified / Radeon 8060S: FP8 → Q8_0 conversion (83.4G, 781 tensors) clean with metadata verified, Q8_0 load and coherent generation under -ngl 99, an 18K-token long-context needle retrieved, and a quant A/B (wikitext-2, 24 chunks) giving PPL 5.325 Q8_0 / 5.377 IQ4_XS-mix / 5.404 Q4_K_M at 17.7–19.9 t/s decode and 211–243 t/s prefill on ROCm.

To be straight about what the slimmed branch has and has not been through: it is compile-gated, not load-gated. I no longer have the Puzzle GGUF or the HF checkout on this machine, so I have not re-run a conversion or a load smoke since the split. What is verified since: the tree builds clean, every commit builds individually (bisectable), the converter is lint- and syntax-clean at every commit, the model code has zero nextn references left, and the converter emits exactly the 88 trunk blocks the loader expects. It has also been rebased repeatedly onto master without conflicts, most recently onto the current tip. Re-verified on the current tip: builds clean, converter flake8-clean, and the mtp.* handling above is fixed and covered by the class docstring.

GGUFs and a reusable imatrix: https://huggingface.co/YanissAmz/Nemotron-3-Puzzle-75B-A9B-GGUF

A separate converter extension to Nemotron-3-Super-120B has been proposed by another contributor in the comments below; it is small and independent, and best as its own PR.

Note unrelated to this diff: Vulkan (RADV gfx1151) decode collapses on 4-bit MoE quants (mul_mat_id path) while Q8_0 is fine — ROCm/HIP is the fast backend for this model on Strix Halo. Possibly worth a separate issue.

Update — per-layer accessors

The scalar n_ff_exp / n_expert_used fields are gone. Both values are now read through get_key_or_arr into per-layer arrays and exposed as n_ff_exp(il) / n_expert_used(il), built exactly like the existing n_head_kv(il) and n_ff(il): array lookup, GGML_ABORT out of range, il defaulting to 0 so uniform call sites stay a plain n_ff_exp(). The n_expert_used validation now checks the maximum across layers.

Verified locally on DeepSeek-V4-Flash and Qwen3.6-35B-A3B (expert counts match the GGUF metadata, both generate). The per-layer path was checked by injecting array-valued keys into the metadata shard: a wrong per-layer n_ff_exp is caught by check_tensor_dims, and an out-of-range entry on the last layer trips the validation assert. The Puzzle model itself was not re-run here — the conversion was verified in review.

Review feedback of 09-02 is applied in full: _impl is gone in favour of the _arr reads, the accessors abort out of range instead of falling back, the per-key required flags are restored where they were load-bearing, and the remaining call sites that still carried the old column alignment (afmoe, bailingmoe 1/2/3, cohere2moe, deepseek/2ocr/32/4, dots1, and the n_expert_used line in llama-model.h) now match the suggested form. Rebased onto current master, no conflicts; builds clean.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES — AI-assisted. I scoped the architecture, made the design decisions, reviewed every diff, and validated everything end-to-end on my hardware (Strix Halo 128GB, Vulkan/ROCm). I take full responsibility for the code and remain available for any requested change or test.

ggml-gh-bot · 2026-07-08T14:00:07Z

Hi @YanissAmz, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.
AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.
Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

YanissAmz · 2026-07-12T12:14:21Z

Regarding the automated flags above:

PR template: the description follows the current template — Overview, Additional information, and the Requirements section are all present and filled in.
AI usage disclosure: explicitly disclosed as YES in the Requirements section, with a description of how assistance was used. The code was designed, reviewed, and validated end-to-end by me on real hardware (Strix Halo 128GB, ROCm/Vulkan), and I take full responsibility for every change.
PR size: the quantize wiring is already split out into a separate branch to keep this PR focused. As noted in the description, I'm happy to split further into two stacked PRs — (a) per-layer MoE arrays + converter, (b) MTP support — if maintainers prefer.

The branch has been rebased onto current master (clean rebase, builds fine). Happy to run any additional test on this hardware on request.

rgex · 2026-07-12T20:37:51Z

Is this also working with Nemotron 3 Super 120B?

YanissAmz · 2026-07-13T12:45:01Z

Super 120B should keep working, it was already supported before this PR (both are 88 layers, the PR just adds a check on the per-layer top-k to tell Puzzle apart — Super is uniform so it still resolves to 120B.A12B). I only tested Puzzle though, don't have a Super GGUF around. If you try it on this branch let me know.

YanissAmz · 2026-07-14T19:36:02Z

Some adoption signal: community GGUF quants of Puzzle-75B built with this branch are now published — https://huggingface.co/RemySkye/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-GGUF. Their README pins this PR (commit af49ef5) and notes that a build containing it is required to load the files.

Their conversion notes also surfaced a compatibility gap worth folding in: NVIDIA's official BF16 upload names the trunk model.* (model.layers.*, model.embeddings, model.norm_f) where the original checkpoint used the NemotronH-style backbone.*, and it spells the router bias e_score_correction_bias instead of e_score_correction.bias. Pushed 04b3780e7 which normalizes both at the top of NemotronHPuzzleModel.modify_tensors, so the officially distributed checkpoint converts as well — checked that every tensor name in the official index (42,683 keys, MTP head included) resolves through the tensor map after normalization.

ross-rosario · 2026-07-25T23:49:35Z

So.... is this just awaiting a final rebase then a review?

Green-Sky · 2026-07-29T07:10:57Z

In the interest of getting this merged, I suggest you pull out the mtp changes into its own pr. It would slim down the scope of this pr a lot.

Premshay · 2026-07-30T08:21:26Z

Chiming in, since I faced a similar issue and worked on extending the converter to Super.
I built the branch and extended the converter opt-in to NemotronHForCausalLM (Super).
I then converted the official BF16; also ran the A/B on CPU.
Conversion now on HF: https://huggingface.co/Premshay/Nemotron-3-Super-120B-A12B-MTP-GGUF

Validation Hardware: Intel Core Ultra 7 270K Plus (Arrow Lake, 24 cores host / 20 WSL2 , AVX2, no AVX-512), 121 GB DDR5, CPU-only.
Model: official NVIDIA-Nemotron-3-Super-120B-A12B-BF16 , converted to Q8_0 quantized to IQ4_NL (66.4 GiB), head intact
(blk.88/89, nextn_predict_layers=2). Protocol: llama-server, -t 20 -c 8192 -np 1 -fa off,
temp 0, thinking disabled, 256-token generations, 3 reps, idle machine.

build	config	t/s (3 reps)	mean	draft acceptance
this PR (`dbedc9e`)	MTP off	8.72 / 8.83 / 8.95	8.83	-
this PR	MTP on	4.99 / 4.99 / 4.95	4.98	127/380 (0.33)
mamba-rs-rollback-wip (`fc7f6aa`)	MTP off	8.72 / 8.94 / 8.96	8.87	-
mamba-rs-rollback-wip	MTP on	10.92 / 12.37 / 12.80	12.03	~172/246 (0.70)

I confirmed that it is functional but not a speed win without rollback. The mamba-rs-rollback-wip turns it into 36-44% speedup with 0.70 acceptance. This is evidence for the split @Green-Sky suggested.

About the divergence: at temp 0 the MTP-on output differs from MTP-off on both builds (1030-char completion for on, 1055 for off). Is that expected from batched-verify numerics on CPU, or a rollback-WIP issue?

The converter addition for the generic class is small, so I offer the following into the split PR:
Premshay/llama.cpp@dbedc9e...nemotron-super-mtp

I can run further tests if needed.
@YanissAmz and @Green-Sky , Let me know if this helps.

Disclosure: Conversion code and benchmarks assisted by AI (Claude). I directed and verified. Numbers measured on my hardware as described.

disarticulate · 2026-08-05T12:45:20Z

just adding a bump with interest; the model size seems to be well places for 72-128GB hardware, which fits a lot of consumer tech.

Premshay · 2026-08-26T19:14:32Z

Still waiting after 2 months...

disarticulate · 2026-08-29T12:06:46Z

@Premshay do you think Nvidia is planning to purchase ggml/Huggingface to get this pull request merged, seeing how fast everyone worked on the Qwen3.8 next?

jacekpoplawski · 2026-08-29T12:39:50Z

Is there any reason why this PR touches 55 files, including all kinds of models?

* src : add n_expert_used_max function With Commit c61b98b ("model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (#25444)") it is now possible for each layer to have a specific number of experts but there are a few checks that need to be updated to handle this upon model loading. For example: ```console llama_model_load: error loading model: model has expert layers but no expert layers are used ``` And later: ```console /llama.cpp/src/llama-model-loader.cpp:955: GGML_ASSERT(n_ids_used > 0) failed ``` This commit adds the n_expert_used_max function so that these checks can use it. Refs: #25444 (comment) * src : use hparams.n_expert_used_max in llama_model_base::load_hparams * src : use 0 as initial value for n_expert_used_max

* src : add n_expert_used_max function With Commit c61b98b ("model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (#25444)") it is now possible for each layer to have a specific number of experts but there are a few checks that need to be updated to handle this upon model loading. For example: ```console llama_model_load: error loading model: model has expert layers but no expert layers are used ``` And later: ```console /llama.cpp/src/llama-model-loader.cpp:955: GGML_ASSERT(n_ids_used > 0) failed ``` This commit adds the n_expert_used_max function so that these checks can use it. Refs: ggml-org/llama.cpp#25444 (comment) * src : use hparams.n_expert_used_max in llama_model_base::load_hparams * src : use 0 as initial value for n_expert_used_max (cherry picked from commit 9a4843c)

* src : add n_expert_used_max function With Commit c61b98b ("model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (#25444)") it is now possible for each layer to have a specific number of experts but there are a few checks that need to be updated to handle this upon model loading. For example: ```console llama_model_load: error loading model: model has expert layers but no expert layers are used ``` And later: ```console /llama.cpp/src/llama-model-loader.cpp:955: GGML_ASSERT(n_ids_used > 0) failed ``` This commit adds the n_expert_used_max function so that these checks can use it. Refs: ggml-org/llama.cpp#25444 (comment) * src : use hparams.n_expert_used_max in llama_model_base::load_hparams * src : use 0 as initial value for n_expert_used_max

…ggml-org#25444) * hparams: add per-layer n_ff_exp/n_expert_used arrays with scalar-or-array loading G1/G2 infrastructure for variable-per-layer expert FFN size and top-k routing (required for Puzzle-75B which has 5 distinct n_ff_exp values and 7 top-k values across its 40 MoE layers). Design: rename scalar members to _impl suffix (following existing convention), add LLAMA_MAX_LAYERS arrays, add n_ff_exp(il)/n_expert_used(il) accessors with scalar fallback. No new GGUF keys: reuses existing expert_feed_forward_length and expert_used_count keys via get_key_or_arr (scalar -> broadcast, array -> per-layer). - llama-hparams.h: n_ff_exp -> n_ff_exp_impl, n_expert_used -> n_expert_used_impl; add n_ff_exp_arr / n_expert_used_arr arrays; add per-layer accessor declarations. - llama-hparams.cpp: implement n_ff_exp(il) and n_expert_used(il); out-of-range il returns impl safely (shared code, no abort). - llama-model.cpp: central n_expert_used load changed to get_key_or_arr; derive impl as max-of-array for validations and backward compat; zero both new arrays; HunyuanVL override also zeroes n_expert_used_arr. - llama-graph.cpp: aggregation loop in build_moe_ffn uses hparams.n_expert_used(il) so per-layer top-k bounds the ggml_view loop correctly. - All other files: mechanical rename hparams.n_{ff_exp,expert_used} -> *_impl. Scalar arches are unaffected (broadcast fills all array slots with the single value). (cherry picked from commit 269a81e) * nemotron-h: use per-layer n_ff_exp(il) and n_expert_used(il) at MoE call-sites Load n_ff_exp via get_key_or_arr into hparams.n_ff_exp_arr in load_arch_hparams; derive impl as max for existing uniform GGUFs. In load_arch_tensors, compute n_ff_exp_i = hparams.n_ff_exp(i) with fallback to n_ff(i)/n_expert_used(i) for GGUFs that omit expert_feed_forward_length. In build_ffn_layer, pass hparams.n_expert_used(il) to build_moe_ffn so per-layer top-k is used for expert routing selection. All other nemotron-h behaviour (mamba2, attention, shared-exp, latent projection, routed_scaling_factor, expert_weights_norm, sigmoid gating) is unchanged. (cherry picked from commit b1878a1) * arch/*.cpp + gguf-py: mechanical rename n_ff_exp->n_ff_exp_impl, n_expert_used->n_expert_used_impl All non-nemotron arch files continue using the scalar impl member directly. Behaviour is identical: the impl value is the broadcast value from the GGUF scalar. gguf_writer: add_expert_feed_forward_length and add_expert_used_count now accept int | Sequence[int], mirroring add_feed_forward_length, so converters can write per-layer arrays with the same existing GGUF keys. (cherry picked from commit 8f009f5) * convert: support NemotronHPuzzleForCausalLM (per-block MoE config) Parse block_configs/mtp_block_configs into per-layer arrays (scalar-or-array keys), append the MTP [attention, moe] sub-blocks as blk.88/blk.89 with nextn tensors, accept the backbone.* prefix, and register the arch. Also fix a pre-existing undeclared _experts attribute on NemotronHModel. (cherry picked from commit d1a592f) * nemotron-h: distinguish Nemotron 3 Puzzle (75B.A9B) from Super (120B.A12B) Both have 88 layers; the per-layer expert_used_count array (heterogeneous for Puzzle, broadcast-uniform for Super) is the discriminator. (cherry picked from commit f824e09) * convert: accept the official Puzzle BF16 checkpoint's tensor naming The officially distributed BF16 checkpoint (NVIDIA-Nemotron-Labs-3-Puzzle- 75B-A9B-BF16) names the trunk model.* (model.layers.*, model.embeddings, model.norm_f) where the original release used the NemotronH-style backbone.*, and spells the router bias e_score_correction_bias instead of e_score_correction.bias. Normalize both at the top of NemotronHPuzzleModel.modify_tensors so either checkpoint converts; every tensor name in the official index (42683 keys, MTP head included) resolves through the tensor map after normalization. (cherry picked from commit 189b67f) * laguna: use n_ff_exp_impl for the uniform-MoE FFN size Laguna landed after this branch was cut and reads hparams.n_ff_exp as a scalar. This series turns it into a per-layer array with an n_ff_exp(il) accessor, so the three scalar reads no longer compile. Laguna is a uniform MoE, so point them at the scalar fallback n_ff_exp_impl, same as deepseek2/qwen3moe/gemma4 in this series. No behaviour change. (cherry picked from commit dbedc9e) * arch: extend the n_ff_exp/n_expert_used rename to archs added upstream kimi-k3, dflash, bailingmoe3, deepseek4, granite-swa and the nemotron-h MTP block still referenced the scalar fields by their old names. n_ff_exp and n_expert_used are accessors now, so those reads no longer compile; point the non-per-layer archs at the _impl scalars and use the indexed form where the call site is per-layer. * convert: keep Puzzle opted out of the NemotronH MTP export path ggml-org#26725 added MTP export to NemotronHModel, keyed on num_nextn_predict_layers. Puzzle's config carries that key, but NemotronHPuzzleModel bypasses NemotronHModel.__init__ (its per-block config needs a different setup), so _mtp_bid was never assigned and modify_tensors raised AttributeError on any mtp.* tensor. Puzzle's head is also laid out by mtp_block_configs, not the mtp.layers.* form the base maps. Set _mtp_bid to None, drop mtp.* in filter_tensors, and declare supports_mtp_export = False so --mtp / --no-mtp fail at the CLI. * llama: replace n_ff_exp/n_expert_used scalars with per-layer accessors Follow-up to review feedback: the previous revision kept the scalar hparams fields alongside the new per-layer arrays, which duplicated state that get_key_or_arr already handles by broadcasting a scalar value over every layer. Drop both scalars and expose n_ff_exp(il) / n_expert_used(il) built exactly like the existing n_head_kv(il) and n_ff(il) accessors: they index the array and GGML_ABORT out of range, with il defaulting to 0 so genuinely uniform call sites stay a plain n_ff_exp(). Arch loaders now read both keys through get_key_or_arr over n_layer_all, and the n_expert_used validation checks the maximum across layers instead of a single field. * llama: restore per-key required flags on the expert hparam reads The scalar-to-array conversion passed required=false at every call site, which silently made mandatory keys optional. Each read now carries the same required flag it had before the conversion.

glennhanks · 2026-09-09T00:28:16Z

I think you may have broken loading right at the end, I get this error:
llama_model_load: error loading model: model has expert layers but no expert layers are used

I got a Floating point exception (core dumped) running rocm 7.24, loading error from both rocm and vulkan backend. Please please please help fix this, this model is very interesting. I ran q4_k_m on the old PR I compiled before, it works but quite slow. But shockingly, it is the only model so far that understands a humor story that no model before (not even the closed AIs) was able to appreciate! Mindblowing for me.

…ggml-org#25444) * hparams: add per-layer n_ff_exp/n_expert_used arrays with scalar-or-array loading G1/G2 infrastructure for variable-per-layer expert FFN size and top-k routing (required for Puzzle-75B which has 5 distinct n_ff_exp values and 7 top-k values across its 40 MoE layers). Design: rename scalar members to _impl suffix (following existing convention), add LLAMA_MAX_LAYERS arrays, add n_ff_exp(il)/n_expert_used(il) accessors with scalar fallback. No new GGUF keys: reuses existing expert_feed_forward_length and expert_used_count keys via get_key_or_arr (scalar -> broadcast, array -> per-layer). - llama-hparams.h: n_ff_exp -> n_ff_exp_impl, n_expert_used -> n_expert_used_impl; add n_ff_exp_arr / n_expert_used_arr arrays; add per-layer accessor declarations. - llama-hparams.cpp: implement n_ff_exp(il) and n_expert_used(il); out-of-range il returns impl safely (shared code, no abort). - llama-model.cpp: central n_expert_used load changed to get_key_or_arr; derive impl as max-of-array for validations and backward compat; zero both new arrays; HunyuanVL override also zeroes n_expert_used_arr. - llama-graph.cpp: aggregation loop in build_moe_ffn uses hparams.n_expert_used(il) so per-layer top-k bounds the ggml_view loop correctly. - All other files: mechanical rename hparams.n_{ff_exp,expert_used} -> *_impl. Scalar arches are unaffected (broadcast fills all array slots with the single value). (cherry picked from commit 269a81e) * nemotron-h: use per-layer n_ff_exp(il) and n_expert_used(il) at MoE call-sites Load n_ff_exp via get_key_or_arr into hparams.n_ff_exp_arr in load_arch_hparams; derive impl as max for existing uniform GGUFs. In load_arch_tensors, compute n_ff_exp_i = hparams.n_ff_exp(i) with fallback to n_ff(i)/n_expert_used(i) for GGUFs that omit expert_feed_forward_length. In build_ffn_layer, pass hparams.n_expert_used(il) to build_moe_ffn so per-layer top-k is used for expert routing selection. All other nemotron-h behaviour (mamba2, attention, shared-exp, latent projection, routed_scaling_factor, expert_weights_norm, sigmoid gating) is unchanged. (cherry picked from commit b1878a1) * arch/*.cpp + gguf-py: mechanical rename n_ff_exp->n_ff_exp_impl, n_expert_used->n_expert_used_impl All non-nemotron arch files continue using the scalar impl member directly. Behaviour is identical: the impl value is the broadcast value from the GGUF scalar. gguf_writer: add_expert_feed_forward_length and add_expert_used_count now accept int | Sequence[int], mirroring add_feed_forward_length, so converters can write per-layer arrays with the same existing GGUF keys. (cherry picked from commit 8f009f5) * convert: support NemotronHPuzzleForCausalLM (per-block MoE config) Parse block_configs/mtp_block_configs into per-layer arrays (scalar-or-array keys), append the MTP [attention, moe] sub-blocks as blk.88/blk.89 with nextn tensors, accept the backbone.* prefix, and register the arch. Also fix a pre-existing undeclared _experts attribute on NemotronHModel. (cherry picked from commit d1a592f) * nemotron-h: distinguish Nemotron 3 Puzzle (75B.A9B) from Super (120B.A12B) Both have 88 layers; the per-layer expert_used_count array (heterogeneous for Puzzle, broadcast-uniform for Super) is the discriminator. (cherry picked from commit f824e09) * convert: accept the official Puzzle BF16 checkpoint's tensor naming The officially distributed BF16 checkpoint (NVIDIA-Nemotron-Labs-3-Puzzle- 75B-A9B-BF16) names the trunk model.* (model.layers.*, model.embeddings, model.norm_f) where the original release used the NemotronH-style backbone.*, and spells the router bias e_score_correction_bias instead of e_score_correction.bias. Normalize both at the top of NemotronHPuzzleModel.modify_tensors so either checkpoint converts; every tensor name in the official index (42683 keys, MTP head included) resolves through the tensor map after normalization. (cherry picked from commit 189b67f) * laguna: use n_ff_exp_impl for the uniform-MoE FFN size Laguna landed after this branch was cut and reads hparams.n_ff_exp as a scalar. This series turns it into a per-layer array with an n_ff_exp(il) accessor, so the three scalar reads no longer compile. Laguna is a uniform MoE, so point them at the scalar fallback n_ff_exp_impl, same as deepseek2/qwen3moe/gemma4 in this series. No behaviour change. (cherry picked from commit dbedc9e) * arch: extend the n_ff_exp/n_expert_used rename to archs added upstream kimi-k3, dflash, bailingmoe3, deepseek4, granite-swa and the nemotron-h MTP block still referenced the scalar fields by their old names. n_ff_exp and n_expert_used are accessors now, so those reads no longer compile; point the non-per-layer archs at the _impl scalars and use the indexed form where the call site is per-layer. * convert: keep Puzzle opted out of the NemotronH MTP export path ggml-org#26725 added MTP export to NemotronHModel, keyed on num_nextn_predict_layers. Puzzle's config carries that key, but NemotronHPuzzleModel bypasses NemotronHModel.__init__ (its per-block config needs a different setup), so _mtp_bid was never assigned and modify_tensors raised AttributeError on any mtp.* tensor. Puzzle's head is also laid out by mtp_block_configs, not the mtp.layers.* form the base maps. Set _mtp_bid to None, drop mtp.* in filter_tensors, and declare supports_mtp_export = False so --mtp / --no-mtp fail at the CLI. * llama: replace n_ff_exp/n_expert_used scalars with per-layer accessors Follow-up to review feedback: the previous revision kept the scalar hparams fields alongside the new per-layer arrays, which duplicated state that get_key_or_arr already handles by broadcasting a scalar value over every layer. Drop both scalars and expose n_ff_exp(il) / n_expert_used(il) built exactly like the existing n_head_kv(il) and n_ff(il) accessors: they index the array and GGML_ABORT out of range, with il defaulting to 0 so genuinely uniform call sites stay a plain n_ff_exp(). Arch loaders now read both keys through get_key_or_arr over n_layer_all, and the n_expert_used validation checks the maximum across layers instead of a single field. * llama: restore per-key required flags on the expert hparam reads The scalar-to-array conversion passed required=false at every call site, which silently made mandatory keys optional. Each read now carries the same required flag it had before the conversion.

* src : add n_expert_used_max function With Commit c61b98b ("model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (ggml-org#25444)") it is now possible for each layer to have a specific number of experts but there are a few checks that need to be updated to handle this upon model loading. For example: ```console llama_model_load: error loading model: model has expert layers but no expert layers are used ``` And later: ```console /llama.cpp/src/llama-model-loader.cpp:955: GGML_ASSERT(n_ids_used > 0) failed ``` This commit adds the n_expert_used_max function so that these checks can use it. Refs: ggml-org#25444 (comment) * src : use hparams.n_expert_used_max in llama_model_base::load_hparams * src : use 0 as initial value for n_expert_used_max

* src : add n_expert_used_max function With Commit c61b98b ("model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (#25444)") it is now possible for each layer to have a specific number of experts but there are a few checks that need to be updated to handle this upon model loading. For example: ```console llama_model_load: error loading model: model has expert layers but no expert layers are used ``` And later: ```console /llama.cpp/src/llama-model-loader.cpp:955: GGML_ASSERT(n_ids_used > 0) failed ``` This commit adds the n_expert_used_max function so that these checks can use it. Refs: ggml-org/llama.cpp#25444 (comment) * src : use hparams.n_expert_used_max in llama_model_base::load_hparams * src : use 0 as initial value for n_expert_used_max

…ggml-org#25444) * hparams: add per-layer n_ff_exp/n_expert_used arrays with scalar-or-array loading G1/G2 infrastructure for variable-per-layer expert FFN size and top-k routing (required for Puzzle-75B which has 5 distinct n_ff_exp values and 7 top-k values across its 40 MoE layers). Design: rename scalar members to _impl suffix (following existing convention), add LLAMA_MAX_LAYERS arrays, add n_ff_exp(il)/n_expert_used(il) accessors with scalar fallback. No new GGUF keys: reuses existing expert_feed_forward_length and expert_used_count keys via get_key_or_arr (scalar -> broadcast, array -> per-layer). - llama-hparams.h: n_ff_exp -> n_ff_exp_impl, n_expert_used -> n_expert_used_impl; add n_ff_exp_arr / n_expert_used_arr arrays; add per-layer accessor declarations. - llama-hparams.cpp: implement n_ff_exp(il) and n_expert_used(il); out-of-range il returns impl safely (shared code, no abort). - llama-model.cpp: central n_expert_used load changed to get_key_or_arr; derive impl as max-of-array for validations and backward compat; zero both new arrays; HunyuanVL override also zeroes n_expert_used_arr. - llama-graph.cpp: aggregation loop in build_moe_ffn uses hparams.n_expert_used(il) so per-layer top-k bounds the ggml_view loop correctly. - All other files: mechanical rename hparams.n_{ff_exp,expert_used} -> *_impl. Scalar arches are unaffected (broadcast fills all array slots with the single value). (cherry picked from commit 269a81e) * nemotron-h: use per-layer n_ff_exp(il) and n_expert_used(il) at MoE call-sites Load n_ff_exp via get_key_or_arr into hparams.n_ff_exp_arr in load_arch_hparams; derive impl as max for existing uniform GGUFs. In load_arch_tensors, compute n_ff_exp_i = hparams.n_ff_exp(i) with fallback to n_ff(i)/n_expert_used(i) for GGUFs that omit expert_feed_forward_length. In build_ffn_layer, pass hparams.n_expert_used(il) to build_moe_ffn so per-layer top-k is used for expert routing selection. All other nemotron-h behaviour (mamba2, attention, shared-exp, latent projection, routed_scaling_factor, expert_weights_norm, sigmoid gating) is unchanged. (cherry picked from commit b1878a1) * arch/*.cpp + gguf-py: mechanical rename n_ff_exp->n_ff_exp_impl, n_expert_used->n_expert_used_impl All non-nemotron arch files continue using the scalar impl member directly. Behaviour is identical: the impl value is the broadcast value from the GGUF scalar. gguf_writer: add_expert_feed_forward_length and add_expert_used_count now accept int | Sequence[int], mirroring add_feed_forward_length, so converters can write per-layer arrays with the same existing GGUF keys. (cherry picked from commit 8f009f5) * convert: support NemotronHPuzzleForCausalLM (per-block MoE config) Parse block_configs/mtp_block_configs into per-layer arrays (scalar-or-array keys), append the MTP [attention, moe] sub-blocks as blk.88/blk.89 with nextn tensors, accept the backbone.* prefix, and register the arch. Also fix a pre-existing undeclared _experts attribute on NemotronHModel. (cherry picked from commit d1a592f) * nemotron-h: distinguish Nemotron 3 Puzzle (75B.A9B) from Super (120B.A12B) Both have 88 layers; the per-layer expert_used_count array (heterogeneous for Puzzle, broadcast-uniform for Super) is the discriminator. (cherry picked from commit f824e09) * convert: accept the official Puzzle BF16 checkpoint's tensor naming The officially distributed BF16 checkpoint (NVIDIA-Nemotron-Labs-3-Puzzle- 75B-A9B-BF16) names the trunk model.* (model.layers.*, model.embeddings, model.norm_f) where the original release used the NemotronH-style backbone.*, and spells the router bias e_score_correction_bias instead of e_score_correction.bias. Normalize both at the top of NemotronHPuzzleModel.modify_tensors so either checkpoint converts; every tensor name in the official index (42683 keys, MTP head included) resolves through the tensor map after normalization. (cherry picked from commit 189b67f) * laguna: use n_ff_exp_impl for the uniform-MoE FFN size Laguna landed after this branch was cut and reads hparams.n_ff_exp as a scalar. This series turns it into a per-layer array with an n_ff_exp(il) accessor, so the three scalar reads no longer compile. Laguna is a uniform MoE, so point them at the scalar fallback n_ff_exp_impl, same as deepseek2/qwen3moe/gemma4 in this series. No behaviour change. (cherry picked from commit dbedc9e) * arch: extend the n_ff_exp/n_expert_used rename to archs added upstream kimi-k3, dflash, bailingmoe3, deepseek4, granite-swa and the nemotron-h MTP block still referenced the scalar fields by their old names. n_ff_exp and n_expert_used are accessors now, so those reads no longer compile; point the non-per-layer archs at the _impl scalars and use the indexed form where the call site is per-layer. * convert: keep Puzzle opted out of the NemotronH MTP export path ggml-org#26725 added MTP export to NemotronHModel, keyed on num_nextn_predict_layers. Puzzle's config carries that key, but NemotronHPuzzleModel bypasses NemotronHModel.__init__ (its per-block config needs a different setup), so _mtp_bid was never assigned and modify_tensors raised AttributeError on any mtp.* tensor. Puzzle's head is also laid out by mtp_block_configs, not the mtp.layers.* form the base maps. Set _mtp_bid to None, drop mtp.* in filter_tensors, and declare supports_mtp_export = False so --mtp / --no-mtp fail at the CLI. * llama: replace n_ff_exp/n_expert_used scalars with per-layer accessors Follow-up to review feedback: the previous revision kept the scalar hparams fields alongside the new per-layer arrays, which duplicated state that get_key_or_arr already handles by broadcasting a scalar value over every layer. Drop both scalars and expose n_ff_exp(il) / n_expert_used(il) built exactly like the existing n_head_kv(il) and n_ff(il) accessors: they index the array and GGML_ABORT out of range, with il defaulting to 0 so genuinely uniform call sites stay a plain n_ff_exp(). Arch loaders now read both keys through get_key_or_arr over n_layer_all, and the n_expert_used validation checks the maximum across layers instead of a single field. * llama: restore per-key required flags on the expert hparam reads The scalar-to-array conversion passed required=false at every call site, which silently made mandatory keys optional. Each read now carries the same required flag it had before the conversion.

* src : add n_expert_used_max function With Commit 3615d1b ("model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support (ggml-org#25444)") it is now possible for each layer to have a specific number of experts but there are a few checks that need to be updated to handle this upon model loading. For example: ```console llama_model_load: error loading model: model has expert layers but no expert layers are used ``` And later: ```console /llama.cpp/src/llama-model-loader.cpp:955: GGML_ASSERT(n_ids_used > 0) failed ``` This commit adds the n_expert_used_max function so that these checks can use it. Refs: ggml-org#25444 (comment) * src : use hparams.n_expert_used_max in llama_model_base::load_hparams * src : use 0 as initial value for n_expert_used_max

YanissAmz · 2026-09-11T18:59:51Z

@bartowski1182 the "no expert layers are used" error is fixed by #28323 (thanks @danbev).

@glennhanks the floating point exception is my bug. GGUFs made with the review branch (mine and RemySkye's) divide by zero at load. I'll send a fix, but it only turns the crash into an error: those files need reconverting with the current converter. For now, your old build still works with them.

@Premshay sorry for the silence, and thanks for the benchmarks. They confirm MTP needs partial rollback to pay off on this model, which is what #25691 does. I'll come back to it.

… Puzzle) Cherry-picked upstream ggml-org#25444 renamed scalar hparams.n_expert_used and hparams.n_ff_exp to _impl fields and added per-layer accessors (matching the previous PR ggml-org#26869 rename of the scalar fields). CachyLLama missed a few call sites: - llama_model_n_expert_used (C API) - llama_moe_residency - llama_context stats/selection populators - qwen4exp load_arch_tensors MTP-layer loop (line 276, originally wrote n_ff_exp without the parenthesised accessor; pre-rename it worked because the field existed; post-rename it has to use the method) All bare hparams.n_expert_used / hparams.n_ff_exp references in CachyLLama code now use the accessors.

YanissAmz requested review from a team, CISC and ggerganov as code owners July 8, 2026 13:55

github-actions Bot added model Model specific conversion labels Jul 8, 2026

YanissAmz mentioned this pull request Jul 8, 2026

quantize: wire NVFP4 ftype (experts NVFP4, other tensors Q8_0) #25446

Closed

This was referenced Jul 8, 2026

convert: fix NemotronHPuzzle AttributeError on expert tensors; add array support to add_expert_feed_forward_length rcmorano/ROCmFPX#4

Closed

convert_hf_to_gguf: harden NemotronHPuzzle MTP tensor remapping rcmorano/ROCmFPX#5

Closed

YanissAmz force-pushed the puzzle-port branch from af49ef5 to 531a8a3 Compare July 12, 2026 12:14

YanissAmz force-pushed the puzzle-port branch from 531a8a3 to c91d415 Compare July 12, 2026 13:32

YanissAmz force-pushed the puzzle-port branch from 04b3780 to b566325 Compare July 14, 2026 20:30

YanissAmz force-pushed the puzzle-port branch from b566325 to dbedc9e Compare July 26, 2026 10:44

YanissAmz force-pushed the puzzle-port branch 2 times, most recently from 497ec49 to adeff9b Compare July 31, 2026 10:32

YanissAmz changed the title ~~model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support with MTP~~ model: add NVIDIA Nemotron-3-Puzzle-75B-A9B (NemotronHPuzzle) support Jul 31, 2026

YanissAmz force-pushed the puzzle-port branch 2 times, most recently from 1096e1c to 4767ffb Compare August 23, 2026 15:21

YanissAmz force-pushed the puzzle-port branch from 10e7062 to 85d300a Compare September 1, 2026 11:49

CISC reviewed Sep 3, 2026

View reviewed changes

Comment thread src/llama-model.cpp

danbev mentioned this pull request Sep 3, 2026

src : add n_expert_used_max function #28323

Merged

loenhart mentioned this pull request Sep 3, 2026

Misc. bug: SIGFPE (integer divide-by-zero) loading nemotron_h_moe GGUFs whose NextN/MTP blocks have zeroed per-layer expert arrays #28345

Closed

CISC mentioned this pull request Sep 4, 2026

Feature Request: Support per-layer active expert count (top-k) for MoE architectures #28380

Closed

4 tasks done

ggerganov mentioned this pull request Sep 4, 2026

llama.cpp : bump version to 0.4.0 #28386

Merged

YanissAmz mentioned this pull request Sep 11, 2026

nemotron-h : guard the expert FFN size fallback against a zero divisor #28779

Merged

harveyff mentioned this pull request Sep 15, 2026

[UPDATE][llamacppllmbasev3][1.2.38] Update CUDA engine to b10969 beclab/apps#3544

Merged

1 task

christopherthompson81 mentioned this pull request Oct 22, 2026

Nemotron-H: support the hybrid Mamba-2 text models, their MTP head, and the VL / Omni wrappers christopherthompson81/inference.rs#565

Open

Conversation

YanissAmz commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Update — per-layer accessors

Requirements

Uh oh!

ggml-gh-bot Bot commented Jul 8, 2026

Uh oh!

YanissAmz commented Jul 12, 2026

Uh oh!

rgex commented Jul 12, 2026

Uh oh!

YanissAmz commented Jul 13, 2026

Uh oh!

YanissAmz commented Jul 14, 2026

Uh oh!

ross-rosario commented Jul 25, 2026

Uh oh!

Green-Sky commented Jul 29, 2026

Uh oh!

Premshay commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

disarticulate commented Aug 5, 2026

Uh oh!

Premshay commented Aug 26, 2026

Uh oh!

disarticulate commented Aug 29, 2026

Uh oh!

jacekpoplawski commented Aug 29, 2026

Uh oh!

Uh oh!

glennhanks commented Sep 9, 2026

Uh oh!

YanissAmz commented Sep 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

YanissAmz commented Jul 8, 2026 •

edited

Loading

Premshay commented Jul 30, 2026 •

edited

Loading