[ghazni101]

Checkpoint tokenizer.json files that ship a bare ByteLevel post_processor (every Qwen-family checkpoint) leave eos_id_/bos_id_ unset: ExtractBosEos reads only the TemplateProcessing template. When config.json and generation_config.json also decline to name eos, requests carry no eos id at all — the model emits <|im_end|> and the engine ignores it, generating fake turns to max_tokens. Measured live on Qwen3.5-9B-EXL3-4.00bpw (ROCm gfx1101): <|im_end|> (id 248046) emitted at token 10, request ran to finish_reason=length. Issue: #3364 (local ISSUE-LOCAL-01M3RY20H90NMK37V74EH34Z77, mirrored and synced).

HF resolves tokenizer.eos_token_id from the eos_token/bos_token NAMES in tokenizer_config.json (string or AddedToken object), optionally supplied by special_tokens_map.json — AutoTokenizer on this checkpoint answers 248046 (measured, transformers 5.17.0). FromHfJson now reads both sibling files and fills any id the post_processor left at -1. Template ids keep precedence; add_bos_token/add_eos_token stay unhonoured (HF ignores them on load — pinned); tokenizer_config.json wins over special_tokens_map.json via contains()-gated overwrite, matching from_pretrained kwarg ordering.

Verify: test_bpe 30/30, 1020 assertions (run in the ROCm toolchain image; the host lacks cmake so the checker is not host-reproducible). Mutation check: reverting the precedence fix fails the new override subcase with EosId -1 vs 19. Live verification: serving container rebuilt on this change returns finish_reason=stop with clean reasoning/content split, where the unfixed build ran to length.

Out of scope: add_bos/add_eos encode-side application (HF does not apply them either); GGUF path untouched (own kv EOS).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: OMP:devin/swe-2 [omp]

[localai-org-maint-bot]

The new sibling-file lookup is skipped for a valid bare relative filename. In FromHfJson, std::filesystem::path("tokenizer.json").parent_path() is empty, so if (!dir.empty()) bypasses the entire EOS/BOS fix even when tokenizer_config.json is in the current directory. Loading the same bytes as "./tokenizer.json" takes the new path instead. For the ByteLevel fixture with eos_token: "<|end|>", the first spelling therefore leaves EosId() at -1 while the second resolves 19.

Please resolve an empty parent as . (or allow the empty directory to participate in the sibling join) and add a regression that loads both path spellings from the fixture directory. This is a public Tokenizer::FromHfJson path, and callers such as the tokenizer example pass the supplied filename directly.

I read the complete diff at d5e6535 against current main fce3673 and checked the callers. This is a static finding: the C++ regression, oracle comparison, and repository gate could not run here because Python/CMake/the C++ compiler are absent. I cannot push a verified repair under the repository's gate-before-push requirement from this environment. @mudler @richiejp

[ghazni101]

Addressed in cacf4fc.

The !dir.empty() guard is removed. Joining a sibling filename onto the empty parent_path() yields the bare sibling filename, which std::filesystem::exists resolves against the CWD — the same directory the bare tokenizer.json was opened from — so both spellings now take the sibling lookup.

Regression added: subcase "a bare filename resolves siblings from the CWD too" loads both tokenizer.json and ./tokenizer.json from the fixture directory and asserts EosId() == 19 on each. Mutation check reproduced your static finding: restoring the guard fails the new subcase with EosId -1 vs 19 on the bare spelling.

Verify: test_bpe 30/30, 1022 assertions, build-cpu on this host.