Misc. bug: llama-server returns HTTP 500 when thinking is truncated by max_tokens (nlohmann json parse error / ill-formed UTF-8) #315

Name and Version

llama-server --version
version: 0.2.0-dev (build 10754, commit 2459f68b5)
built with MSVC 19.44.35229.0 for Windows AMD64

Binary: llama-prism-b10754-2459f68-bin-win-cuda-13.3-x64.zip (with the matching
cudart-llama-bin-win-cuda-13.3-x64.zip), i.e. the current prism fork binaries.
Model: Ternary-Bonsai-2-27B-Abliterated-PTQ1_0.gguf — ftype : PTQ1_0 - 1.75 bpw ternary (group 128), general.architecture = qwen35.

Operating systems

Windows 11 (26100), CUDA 13.4 driver on an RTX 5060 Laptop GPU (8 GiB).

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server.exe -m Ternary-Bonsai-2-27B-Abliterated-PTQ1_0.gguf ^
  --host 127.0.0.1 --port 8080 ^
  -ngl 99 -fa on -c 65536 -b 2048 -ub 512 -np 4 -kvu ^
  --alias ternary-bonsai-2-27b --jinja -lm none ^
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.05 ^
  -ctk q4_0 -ctv q4_0

Note: no --reasoning-budget (unlimited), no --mmproj, no tools, no
response_format, no grammar.

Problem description & steps to reproduce

With thinking enabled, a prompt that asks for a specific output length makes the model
reason for a long time. If the client's max_tokens is small enough that the reasoning
is cut off before the answer starts, llama-server regularly answers HTTP 500
instead of returning a normal completion (or at least an empty content):

$ curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
    "model": "ternary-bonsai-2-27b",
    "messages": [{"role":"user","content":"Explain attention in about 300 words."}],
    "max_tokens": 2048,
    "chat_template_kwargs": {"enable_thinking": true}
  }'

Response (intermittent, 4 out of 5 attempts):

HTTP/1.1 500 Internal Server Error
{"error":{"code":500,"message":"[json.exception.parse_error.101] parse error at line 1,
 column 1484: syntax error while parsing object - invalid literal; last read: '"We need
 to respond to user: \"Explain attention in about 300 words.\" Need produce final about
 300 words? ...","type":"server_error"}}

A second variant of the same failure is an ill-formed UTF-8 error, which points at the
truncation boundary:

[json.exception.parse_error.101] parse error at line 1, column 1574: syntax error while
parsing value - invalid string: ill-formed UTF-8 byte; last read: '"We need to answer
user: \"Explain attention in about 300 words.\" ...'

On the attempts that do not return 500, the same request returns HTTP 200 with
usage.completion_tokens == 2048 (the cap) and content == "" — i.e. the answer is
empty because the whole budget went into reasoning.

What makes it go away

Raising max_tokens fixes it completely. Same prompt, same server, only the budget
changes:

Other observations:

• temperature: 0 with max_tokens: 2048 did not reproduce the 500 (2/2 attempts),
  but reliably returned the empty answer (200, content == "", completion_tokens == 2048).
  So the 500 depends on the sampled text; the empty answer does not.
• With "chat_template_kwargs": {"enable_thinking": false} the same request with
  max_tokens: 1024 returns HTTP 200 normally.
• A prompt without a length requirement ("Explain attention.") with max_tokens: 2048
  returns HTTP 200 with only ~283 reasoning tokens — so this is not "long prompts are
  broken", it is the truncation path.

Hypothesis

Something server-side re-parses the generated reasoning text as JSON after generation
finishes. When the reasoning is truncated at max_tokens, the text is (a) syntactically
incomplete and/or (b) cut inside a multi-byte UTF-8 sequence, so that parse fails and the
whole request becomes a 500. The path that re-parses the text is worth checking — the
last read strings are the escaped reasoning content, which suggests a JSON document
that already contains the reasoning as an escaped string is being parsed a second time.

The invalid string: ill-formed UTF-8 byte variant in particular suggests the decoded
token sequence at the truncation point is not valid UTF-8, so anything that serialises or
re-parses it will fail.

Related issues

• Misc. bug: llama-server responds with error code 500 and "Failed to parse input at pos ..." message when max_tokens is reached ggml-org/llama.cpp#20193 — "llama-server responds with error code 500 ... when
  max_tokens is reached" (closed 2026-03-08). Same shape (500 instead of a
  max_tokens finish when the cap is hit, on a qwen35 model), but that report used
  /v1/messages with a different message (Failed to parse input at pos ...), so this
  looks like a distinct remaining case rather than the same fix.
• chat: new parser should not crash inference ggml-org/llama.cpp#20708 / chat : add content-only fallback for JSON_NATIVE tool parser ggml-org/llama.cpp#20800 — "the new chat parser should not crash inference" /
  "add content-only fallback for JSON_NATIVE tool parser". If the failing parse here is
  the chat parser, a content-only fallback would turn this 500 into a normal response.
• Misc. bug: MAX_REPETITION_THRESHOLD (2000) too low for peg-native tool-call grammars with many tools / large context, causing silent grammar-compile fallback to unconstrained generation #87 — mentions peg-native tool-call grammars; no tools were
  configured in the reproducer above, but it is the closest fork-side parser surface.