[CC-David-CC]

Title

serve: preserve literal </tool_call> in prompts

Issue: Resolves #2026

Summary

Asking the model to print </tool_call> verbatim can produce an empty final response. The tokenizer treats this spelling as a user-defined special token even inside ordinary message text. Protecting it as plain text in the prompt makes the reported reproduction return the requested string.

What changed

• Add TOOL_LITERAL_TAGS with one entry, </tool_call>, to the existing literal-span protection.
• Protect occurrences in messages, assistant history/reasoning, tool results, definitions, and argument values while preserving template-generated tool-call closers as special tokens.
• Add regressions for prompt token IDs, literal model output, all two-part streaming boundaries, real tool calls, explicit stop sequences, and streaming/non-streaming HTTP responses.

This is a standalone fix against main; it does not depend on #1886. No new state machine, retry, configuration option, or engine change is required. Prompts without the target literal retain identical token IDs.

Extra Notes

Validation

The issue's exact prompt was run through the service with a real IQ3_XXS model on Linux/CUDA (RTX PRO 6000 Blackwell). Ten interleaved before/after pairs used seeds 42-51, temperature 0.6, top_p 0.95, top_k 20, thinking enabled, and no tools. Both arms used the PR code; the baseline restored the original literal registry. Raw input/output token IDs were captured.

Result	Original registry	With fix
Exactly </tool_call>	0/10	10/10
Empty final response	9/10	0/10
Incorrect <|im_start|> response	1/10	0/10

The </aaa> and </mcp_tool> controls each passed before and after. The reporter's IQ3_S/Windows/AMD configuration was not tested.

339 tests passed on the standalone PR branch (Windows, Python 3.13.15). Regression command (no model/GPU required):

python -m unittest serve.test_literal_tool_tags serve.test_frontend serve.test_reasoning_tools serve.test_tool_name_space serve.test_server -q

Removing the new registry entry makes 20 prompt-protection subcases fail; restoring it passes them. Existing output parsing already preserves a standalone closer, so the production fix stays in prompt tokenization.

[kntroh]

I have confirmed that it works properly on an IQ3_S/Windows/AMD configuration. (Successful 10 out of 10 times)

[blange48]

Reproduced and fixed on our side: Print exactly this string and nothing else: </tool_call>, temperature 0.6, top_p 0.95, seeds 42-51, thinking at the template default.

	v0.1.43 + our patches	the same + this PR
exactly </tool_call>	0/10	10/10
empty final response	10/10	0/10

Real tool calls still parse (our 3-call tool test passes: 3/3), and decode is unchanged (1 / 4 / 8 clients: 149 / 269 / 393 tok/s, against 144 / 268 / 392 without it).

Rig: 4x RTX 5080 16 GB (PCIe Gen3), i9-10920X, 123 GiB RAM, Linux, CUDA sm_120. Qwen3.8-Flash-Next GSQ-RCO IQ3_S, --layer-split 12,24,36 --trim-stage-weights --batch 8 --batch-groups 4, MTP --spec 4, 262K, int8 KV (--kv-resident 32768), vision on, conversation cache 16 slots. Base: v0.1.43 plus a few patches. Each variant runs on a fresh engine; this PR is the only difference.

[CC-David-CC]

teamwork makes the dream work

[blange48]

For the record: we now run v0.1.43.1, which includes this fix (#2026 in the release notes). It reads the same on our rig: Print exactly this string and nothing else: </tool_call> returns exactly </tool_call> in 10/10 runs, temperature 0.6, seeds 42-51, against 0/10 (10 empty answers) on v0.1.43. Real tool calls still parse (OpenAI and Anthropic formats, 3/3). Rig: 4x RTX 5080 16 GB (PCIe Gen3), i9-10920X, 123 GiB RAM, Linux, CUDA sm_120; Qwen3.8-Flash-Next GSQ-RCO IQ3_S, --layer-split 12,24,36 --trim-stage-weights --batch 8 --batch-groups 4, MTP --spec 4, 262K, int8 KV (--kv-resident 32768), vision on, conversation cache 16 slots / 24 GiB.

[CC-David-CC]

I have made a number of changes about tool calls this morning:
this is an experimental run ahead fork of the features i am supporting publicly
https://github.com/CC-David-CC/Public_Context_COW

the goal is to upstream to strata but it is EXPERIMENTAL !!

meaning i didn't ask and made unilateral changes

bbuuttt
the tool calling is slightly better

so come take whatever

[jy02375437-max]

Same family reproduces for a different user-defined literal, on the released 0.1.43.1 build (no PR code): </tool_response> returns an empty final response, while the other five of our tokenizer's six USER_DEFINED literals are fine.

Rig: RTX 4090D 48 GB, Windows 11, IQ3_XXS native pack, engine 0.1.43.1 (release zip), --vision. The #2026 prompt shape, one request per literal, 3 reps each, temperature 0.6 / top_p 0.95 / top_k 20:

literal	exact	empty
<tool_call>	3/3	0/3
</tool_call>	3/3	0/3
<tool_response>	3/3	0/3
</tool_response>	0/3	3/3
<think>	3/3	0/3
</think>	3/3	0/3

Raw shape of the failing one: finish_reason: stop, content: null, completion_tokens: 15, reasoning starts The user is asking me to output a specific string verbatim: — the turn closes right there, like the original report.

I read our pack's token_type.json: exactly 6 type-4 (USER_DEFINED) tokens, the six above. So on this build </tool_call> is covered (0.1.43.1) and the thinking markers were covered earlier, but </tool_response> still empties a reply.

Question: would you or the maintainer extend the literal protection to every USER_DEFINED literal? I noticed the generic form in the Public_Context_COW tree ( _literal_tags(controls) marking all CONTROL/USER_DEFINED literals) — if that is the direction, it closes the whole family at once, including any future additions to the vocabulary. Happy to re-run this table (6 literals x 3 reps, ~20 lines against /v1/chat/completions) on any build; it is now part of our own gate so we will notice regressions either way.

[CC-David-CC]

let me look into what it will take to implement before i commit to it

[qiuzan2001]

For triage: #2026 is closed and main fixed it another way in v0.1.43.1: a683000 ("a quoted </tool_call> in a message is text, not the end-of-call token"), then 62185aa (the same for <tool_call>, <tool_response>, </tool_response>) and a0e577d (tests), as blange48 noted above. This branch now conflicts with main in serve/frontend.py, and its TOOL_LITERAL_TAGS covers only </tool_call>, which main's version already includes, so it looks superseded.