[martin-cimmino]

Overview

This PR adds support for domyn/Domyn-Small-v1.0, fixing issues in both the vocab decoder and the Nemotron model loader/converter.

Vocab decoder: without a dedicated pre-tokenizer type, the current code routes the Domyn-Small tokenizer through the generic GPT-2 BPE decode path, which cannot handle Metaspace-encoded spaces (▁ U+2581) and produces corrupted output for every space-prefixed token. This PR adds LLAMA_VOCAB_PRE_TYPE_DOMYN_SMALL = 52 in llama-vocab.h and a new branch in llama-vocab.cpp that sets escape_whitespaces = true. Domyn-Small shares the same regex rules as minerva-7b (same tokenizer family), but we kept that path untouched and added a new key.

Nemotron model loader: create_tensor_qkv was called with n_embd for the Q-projection width and layer.wo was shaped as {n_embd, n_embd}. For a 48-head model like Domyn-Small this causes a tensor shape mismatch crash on load. Fixed to use n_embd_head_k * n_head explicitly for both.

Nemotron converter: the formula int(rot_pct * n_embd) // n_head produces 42 for a 48-head model instead of the correct 64. Fixed to read head_dim from the model config (falling back to n_embd // n_head if absent). Additionally, NemotronModel.set_vocab() unconditionally called _set_vocab_sentencepiece(), crashing when no tokenizer.model file is present. Added a fallback to _set_vocab_gpt2() for BPE-only tokenizers, and registered the Domyn-Small tokenizer hash in get_vocab_base_pre().

Additional information

Tested with domyn/Domyn-Small-v1.0 (BF16 GGUF, 45/45 CTest pass).

Note that this PR only affects models that already have tokenizer.ggml.pre = "domyn-small" in their GGUF.

Hugging Face model link at https://huggingface.co/domyn/Domyn-Small-v1.0

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, I used Claude Code to help locate relevant code paths and draft this PR description. All changes were reviewed and tested by me.

[martin-cimmino]

I think there was a mistake in classifying this PR as AI generated

[CISC]

Note that this PR only affects models that already have tokenizer.ggml.pre = "domyn-small" in their GGUF.

And how would they get that? :)

This value needs to be introduced at conversion, does this model have a unique tokenizer chkhsh?

[martin-cimmino]

Yes, the "domyn-small" string is written during conversion by the companion PR at #23391.

chkhsh = 6e4d72c6e5260b90606e26a0931fa15e4b42ec6e97719d12919f2b1f93478262
That hash is unique to the Domyn-Small-v1.0 tokenizer.

Happy to combine them into a single PR if that's preferred.

[CISC]

Happy to combine them into a single PR if that's preferred.

Yes, makes no sense to separate them, one does not work without the other.

[martin-cimmino]

@CISC Done, I've also updated the PR description.

[martin-cimmino]

@CISC friendly ping on this one when you have a moment 🙏 I've combined the two PRs as you suggested and the branch is ready for another look. No rush — just flagging it in case it got buried. Happy to make any further changes.

[CISC]

@CISC friendly ping on this one when you have a moment 🙏 I've combined the two PRs as you suggested and the branch is ready for another look. No rush — just flagging it in case it got buried. Happy to make any further changes.

It's in my queue, however this one requires checking that we do not break any nemotron models, so might be in the queue a while.

[Jeronymous]

@martin-cimmino @CISC could this get rebased and merged? I just hit the exact bug it fixes and opened #30518: another public Nemotron model won't load (with a head_dim that doesn't match hidden_size/num_heads), involving the same attn_q shape mismatch as this PR solves. It would be a shame to let it stall since the fix is already here.

this one requires checking that we do not break any nemotron models

For what it's worth I went through the diff and the regression surface looks tiny: for standard Nemotron (head_dim = hidden/n_heads), n_embd_head_k * n_head == n_embd, so the Q/wo shapes are unchanged and K/V already used n_embd_gqa; the rope change falls back to n_embd // n_head, which gives the same rope_dimension_count as before; and set_vocab only adds an else-branch for models without a tokenizer.model, so sentencepiece ones are untouched. The only things that actually change are for the decoupled-head_dim case that can't load today.

[CISC]

Sorry for the delay.

@martin-cimmino Can you rebase and update convert_hf_to_gguf_update.py with an entry for the model?

[martin-cimmino]

Hello @CISC, I rebased and updated the convert_hf_to_gguf_update.py as suggested.

However, while debugging and testing the generation I noticed something I overlooked in the past changes. The old version tokenized wrong, It used GPT-2 byte-level BPE but Domyn-Small is SentencePiece-style BPE. Basically, spaces came out as junk tokens, newlines were dropped, and <0x0A> appeared literally in the output. So I decided to reuse the existing Sarvam/Gemma4 SPM-style BPE path with two small additions: ▁ is added only at the start of the input, matching HF's Metaspace prepend_scheme: "first" and llama_detokenize strips that leading space again.

Also, I compared master vs this PR for Nemotron-Mini-4B checking that the converted GGUFs are byte-identical, as well as perplexity and generated output.

This should fix #30518, however @Jeronymous it would be good if you test your model on this branch.

[Jeronymous]

This should fix #30518, however @Jeronymous it would be good if you test your model on this branch.

Thanks @martin-cimmino.

I tested my model on this branch. With the loader fix it now loads and generates correctly.
Also no regression on standard Nemotron (I tested Luciole-1B-Instruct-1.1 and Luciole-23B-Instruct-1.1).

Concerning the conversion, I do it on a special fork (because of our special tokenizer) and I could test that your fix on the rope dimension works well for me as well.

[CISC]

Please add TODOs for prepend_scheme for other tokenizers.

[CISC]

@danbev gentle ping

[ggerganov]

Needs rebase

[martin-cimmino]

Needs rebase

Rebased. I moved Domyn-Small's pre-type number to 63 because Flint took 62.