common : add LLM-jp-4.1 Harmony dialect handler (draft for review)

[e-mon]

Summary

Adds a dedicated chat-format handler for LLM-jp-4.1 (llm-jp/llm-jp-4.1-*-thinking), a model that uses the GPT-OSS (Harmony) format with two dialect differences:

1. Its SentencePiece-style tokenizer emits a leading space after every special token, so the model's output reads <|channel|> analysis<|message|> ... / <|start|> assistant to=functions.X<|channel|> commentary <|constrain|> json<|message|> {...} instead of the unspaced GPT-OSS form. With the GPT-OSS handler, llama-server returns 500 "The model produced output that does not match the expected peg-native format" for tool requests and 400 "Failed to initialize samplers" for tool_choice: required.
2. Parallel tool calls are rendered as consecutive assistant messages, every call but the last closed by <|end|> and the last one by <|call|>.

The template declares the dialect on its first line ({#- chat_format=llm-jp-harmony-v1 -#}), which is what the handler selection keys on, so no other template is affected.

Design

• common/parsers/llm-jp-harmony.cpp: a self-contained handler. Prompt handling (message delimiters, thinking tags, continuation, the <|return|> replacement) is that of common_chat_params_init_gpt_oss; the PEG is the GPT-OSS one with the two differences built in. The gpt-oss-20b-specific rules (stray commentary prefix, unsolicited builtin calls) are not carried over: they never occur in this model's output (0 of 400 BFCL generations). The GPT-OSS handler is not modified; the only additions outside the new file are the declaration in parsers.h, the source list entry, the selection branch in common/chat.cpp, the template, tests and docs.
• Special tokens in headers are followed by [ ]*; <|message|> by at most one space, so intentional leading whitespace in a message body survives (" padded" → " padded"). Plain spaces are used rather than p.space() because the GBNF space rule allows a single space, while <|constrain|> json carries the template's space plus the tokenizer's.
• Parallel calls: tool_choice (end start tool_choice)* inside the trigger rule when parallel_tool_calls is set, so the lazy grammar covers every call; with the flag off the grammar constrains the model to one call, as for other formats.
• Lazy-grammar trigger patterns are the GPT-OSS ones with \s* after <|start|> / <|channel|>.
• Selection in common/chat.cpp before the GPT-OSS check; template added to models/templates/; docs rows added.

Tests

• tests/test-chat.cpp: new LLM-jp-4.1 block (final channel, one-space body rule, reasoning + final, partial reasoning, tool call in role/channel header with <|constrain|> json, parallel calls with <|end|>, structured output). ./build/bin/test-chat → [chat] All tests passed! (the peg_tester also validates that the generated grammar accepts each input).
• E2E with the released llm-jp/llm-jp-4.1-8b-thinking-gguf (Q4_K_M) on llama-server (temperature 0, template embedded in the GGUF, no --chat-template-file):

case	result
reasoning + final	reasoning_content and content separated; no leading space in content
single tool call (tool_choice: auto)	get_weather {"city": "Tokyo"} (was 500)
parallel tool calls (parallel_tool_calls: true)	3 calls returned for a prompt asking for three cities at once
tool_choice: required	tool call returned (was 400)
response_format: json_schema	constrained JSON returned
streaming	reasoning_content deltas, then content
tool-result continuation (single, and 2 parallel results attributed by position)	final answer uses the results

Run on Linux (Ubuntu 22.04 on WSL2, RTX 3090, CUDA 12.6 build of this commit): test-chat passes and all cases pass. The same cases also pass on macOS (Metal) with the pre-rebase build of the handler (identical parser code).

Notes

• The template in models/templates/ is the released llm-jp/llm-jp-4.1-8b-thinking chat template (identical to the one embedded in the released GGUFs); the chat_format=llm-jp-harmony-v1 declaration on line 1 is part of the official template (hf/ver4.0: declare the chat format in alpha_1.1's template llm-jp/llm-jp-tokenizer#32, benchmarks? ggml-org/llama.cpp#34).
• Only chat completions are affected; the raw /completion output keeps the tokenizer's spaces.
• LLM-jp-4.0 templates are not covered (they are structurally identical to the GPT-OSS template and do not carry the declaration).

Differential check against the official parser on BFCL prompts

400 BFCL v3 prompts (simple / multiple / parallel / parallel_multiple, 100 each, drawn at random with a fixed seed from the 1,000 non-live AST prompts) were run through llama-server (this branch, CUDA, temperature 0, 4096-token limit, -v + return_tokens: true so that the chat response exposes the generated token ids). The very same token ids were then parsed by the official reference parser (HarmonyMessageParser from the llm-jp-vllm plugin) and compared with what the handler returned:

	agree
tool calls (names + JSON arguments)	400 / 400
reasoning_content	400 / 400
content	400 / 400
special-token leaks / leading spaces	0
invalid tool names	0 (the grammar enforces the declared names)

Outputs covered single calls, 2–4 parallel calls separated by <|end|>, text answers without a call, and 11 generations cut off by the token limit. With the official bfcl-eval AST checker the same outputs score 83.0% (86 / 85 / 86 / 75 per category; 95% CI 79.0–86.4%); every miss is a model-side argument or count error, a text answer, or a truncation, none is a parsing error.

Precedents

Self-contained handlers added for one model family without touching the others: ggml-org#19931 (GigaChat v3/3.1, by the model vendor), ggml-org#24615 (Cohere2MoE), ggml-org#26210 (MiniMax M3), ggml-org#21418 (Gemma 4); the per-model file layout comes from ggml-org#27764, and ggml-org#20393 is the PEG rework of the GPT-OSS handler this one mirrors.