[tkran2]

Overview

The Tekken pre-tokenizer is now used by pixtral rather than Llama 3's. The reason for this is because the pixtral hash was from the community Pixtral copy, and that utilized the regex of Llama 3. The Tekken regex is used by the official tekken.json from Mistral, Nemotron 3 and 3.5, and Ministral-3, which is why those GGUFs were splitting text incorrectly (as an example, Hindi दी was 2 tokens rather than 1).

Fixes #30894

Additional information

Hugging Face PR link: https://huggingface.co/ggml-org/vocabs/discussions/3

The Nemotron test fails on master on Thai/Hindi and passes with the fix. All 15 test-tokenizer-0 tests now pass. This affects the line next to the one changed by #30940, so one of the two might need a rebase.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. Claude wrote the change and made the test vocab. I reviewed the change and ran the tests. I also wrote the text for the PR.