Skip to content

Literal </think> quoted in reasoning yields empty stop or leaked reasoning (v0.1.37) #537

Closed
Closed
Literal </think> quoted in reasoning yields empty stop or leaked reasoning (v0.1.37)#537

Description

@fenrir-labs76
fenrir-labs76
opened on Oct 2, 2026

Literal </think> quoted in reasoning yields empty stop or leaked reasoning (v0.1.37)

Summary

Strata itself mishandles a literal </think> in ordinary user text and generated
reasoning. A request to quote that exact string returns HTTP 200 with
finish_reason: "stop" and no answer, well below max_tokens. A separate
request asking about its HTML-escaped form can split reasoning_content in
the middle of a quote: reasoning then appears in content and the actual
reasoning delimiter can leak into the answer. Reproduced with direct
/v1/chat/completions, with and without streaming; OpenCode is not required.

Version / configuration

  • Latest stable upstream source: v0.1.37, tag commit
    db4f91a1171d697928b0d2e1f50ef95e25559d4c (unmodified source).
  • Locally built from official Dockerfile with CUDA base pinned to a digest;
    resulting image ID sha256:c54ffff355c5f7b16807fbcd2c432b872022420f81e43aae5d1919b7056c73a1.
    Previous unmodified v0.1.33 image also reproduced it.
  • Qwen3.8-Flash-Next IQ3_S model, model ID
    qwen3.8-flash-next-iq3_s; RTX 5090; text-only, context 262144, INT8 KV,
    single serving slot. Live /props reports chat-template SHA256
    12827f24b742ea4e80cdc12dbcf9622227056b9f797252a3149263d4f9aaadce.
  • Explicit max_tokens: 512; no request stop field, no proxy stop sequence;
    default reasoning on. One request at a time. No patched image or model.

Minimal direct reproduction

Use Strata's own local OpenAI-compatible endpoint; substitute your local port
if necessary. No OpenCode, auth, extensions, tools or external data required.

curl -sS --max-time 30 http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-flash-next-iq3_s","stream":false,"max_tokens":512,"messages":[{"role":"user","content":"Quote this exact literal string, then explain why it appears in some chat transcripts: </think>"}]}'

curl -sS -N --max-time 30 http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-flash-next-iq3_s","stream":true,"max_tokens":512,"messages":[{"role":"user","content":"Quote this exact literal string, then explain why it appears in some chat transcripts: </think>"}]}'

Expected: complete answer quoting </think> and explaining that it is a
reasoning delimiter; reasoning_content remains separate and
finish_reason: "stop" means the answer actually ended. Actual: non-stream
response has message.content: null, message.reasoning_content ends
... Quote this exact literal string, then explain why it appears in some chat transcripts: ,
and finish_reason: "stop". SSE emits the same reasoning prefix, final chunk
delta: {}, finish_reason: "stop", then [DONE]; zero answer deltas. HTTP 200.
Output budget was not exhausted.

Second direct input isolates a misclassified generated quoted tag even when
the user does not send a raw delimiter:

curl -sS --max-time 30 http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-flash-next-iq3_s","stream":false,"max_tokens":512,"messages":[{"role":"user","content":"What does the HTML entity &lt;/think&gt; represent? Please write its literal text and explain."}]}'

Expected: private reasoning, then a concise answer explaining that the
entities spell </think>. Actual example: reasoning_content ends
... so literal text is while content begins
. It represents closing tag? In HTML, &lt; and &gt; are character references ...;
later content includes another literal </think> and a repeated answer.
The SSE path also switches from delta.reasoning_content to delta.content
at the first quoted tag.

A normal control prompt What is 1+2+4+8+16? Give sum and formula briefly.
returns separate reasoning and final answer 31 on both APIs.

Multi-turn control: user What about 1+2+4+8+16+....+n; assistant message
Powers of 2: sum = 2n - 1.\n</think>\nGeometric series, ratio 2: 2n - 1.;
user why did you write </think>. It can start content with reasoning like
". Need understand context..., return no answer, or end with length after
misclassifying the prior assistant message. This is supplementary; single-turn
cases above isolate the defect without prior history.

Observed frequency / boundary

Three serial repetitions per case per mode against each release, same prompts
and output cap:

Release Input / outcome JSON SSE
0.1.33 Ordinary math: correct final answer 3/3 3/3
0.1.37 Ordinary math: correct final answer 3/3 3/3
0.1.33 Literal quote: stop with empty final answer 3/3 3/3
0.1.37 Literal quote: stop with empty final answer 3/3 3/3
0.1.33 Escaped tag: reasoning starts in answer content 3/3 3/3
0.1.37 Escaped tag: reasoning starts in answer content 3/3 3/3

The two-turn case varied across repetitions; not claimed deterministic. A
follow-up with chat_template_kwargs.enable_thinking=false still truncated
literal-tag quotation on v0.1.33, so merely turning thinking off was not a
complete workaround.

Source inspection: confirmed vs hypothesis

At the v0.1.37 commit, serve/frontend.py:508 calls
self.buf.find(THINK_END) during reasoning and transitions permanently to
content at that first generated </think>. serve/server.py:1335-1336 renders
the chat template then encodes it with parse_special=True;
serve/server.py:1475 checks configured end-of-turn token IDs. Live tokenizer
encodes literal </think> as ID 248069, even with parse_special=False;
the configured end-of-turn IDs are different. These facts explain how a
quoted tag can collide with the parser's boundary. The exact reason for
the early backend stop (including which generated token ended the turn) is
not proven by these HTTP captures; please do not assume quantization,
OpenCode rendering or an OpenCode stop list is responsible.

Related but distinct: #210 fixed delimiters inside tool parameters in
v0.1.28. This report covers reasoning's </think> boundary and an early
backend stop, still present in the official v0.1.37 build. #530 concerns
finish_reason: "length" from exhausting a thinking budget; the empty-answer
quote here returns stop after a short response.

Activity

  1. fenrir-labs76 commented on Oct 2, 2026

    @fenrir-labs76
    fenrir-labs76
    on Oct 2, 2026
    Author

    Clarification: this is not only a prompt containing a raw </think>.

    The second direct repro in the issue sends no literal </think> in its user message. It sends &lt;/think&gt; and asks the model to write the decoded string. In v0.1.37, the model generates a quoted literal </think> inside its own reasoning. The server emits reasoning_content ending ... so literal text is , then content begins . It represents closing tag? In HTML, ... — still reasoning, now misclassified as answer. Its later, real reasoning-close </think> appears visibly in content, followed by a second answer. That direct response ends with finish_reason: "stop". The same channel switch occurs in SSE. We saw reasoning leak on 3/3 JSON and 3/3 SSE repeats, without a raw delimiter in the user prompt.

    Separate behavior: when the user does type raw </think> and asks it to quote that string, 3/3 JSON and 3/3 SSE runs returned finish_reason: "stop" with no answer. These observations demonstrate both self-generated-tag misclassification and premature backend completion. They do not prove which generated token caused the empty-answer stop; raw token IDs were not captured. Please investigate both paths, rather than only escaping user input or hiding visible tags.

  2. beatcollector commented on Oct 2, 2026

    @beatcollector
    beatcollector
    on Oct 2, 2026

    Tagging on to your issue as an extra data point just because I reproduced it with thinking off as well.

    Unfinished tool call is silently discarded at finish() -> empty content + clean stop

    Engine: Strata 0.1.31 (commit 9259cad), serve/frontend.py OutputParser
    Affects: any agent harness that offers tools and reads OpenAI-compatible responses
    Severity: high for autonomous agents — a stalled turn looks like a completed one

    Symptom

    When generation ends while the model is mid-tool-call (it emitted the call-open marker, or an
    announcement plus partial arguments, then EOS), finish() discards the unfinished phantom call.
    The client receives:

    • content: empty
    • tool_calls: absent
    • finish_reason: "stop" (clean)
    • reasoning content present when thinking mode is on

    A harness that promotes reasoning to the final answer (as Hermes does) then reports a stalled,
    reasoning-only turn as a completed response. Observed repeatedly with both thinking modes on and
    off; every occurrence in our session was triggered by generation touching a call-open or
    call-close marker sequence mid-stream.

    Reproduction (unit level, GPU-free)

    Import the parser's own constants — hand-assembled lookalikes never enter call-state and prove
    nothing:

    from serve.frontend import OutputParser  # CALL_START / CALL_END as defined in the module

p = OutputParser(thinking=True)
chunks = [CALL_START, "terminal", PARAM_START, "command"]   # generation dies mid-call
events = []
for c in chunks:
    events.extend(p.feed(c))
events.extend(p.finish())
# Observed: no tool_call event, content total == 0 chars. The announced call body vanishes.

    Live repro against the server: any prompt that makes the model announce a tool call and then end
    before completing it (with tools present in the request). Without tools, nothing triggers —
    the failure only exists on the tools axis.

    Expected behavior

    Salvage instead of discarding: flush announced-but-unfinished call text back as plain content
    so the harness can at least see the truncation and re-prompt. (Optionally also surface a partial
    call in a diagnostic field.) A discarded phantom call with a clean finish reason is indistinguishable
    from a legitimate empty answer — that is the worst possible signal for an autonomous loop.

    For reference: our local hotfixes on this tree

    With both applied, the same repro prompt returns non-empty content in thinking-on and thinking-off
    modes; complete calls still parse to proper tool_calls (no happy-path regression).

    Contrast: vLLM's parser does not discard

    Tiny probes against vLLM (qwen-family, Hermes-style tags) show an unfinished call structure
    surfacing as plain content with finish_reason=stop, and a complete call parsing correctly. So the
    silent-discard behavior appears specific to this engine family — worth confirming in
    serve/frontend.py finish() handling of an open call state.

    Harness-side mitigation already shipped (Hermes)

    Hermes now classifies text stops ending mid-construct (dangling marker prefix, unclosed fence,
    action verb on a dangling colon) as stalls and re-prompts with a bounded budget — but that only
    recovers the turn; it cannot recover the model's intended call. Engine-side salvage is the fix at
    the source.

  3. Niko1221 commented on Oct 3, 2026

    @Niko1221
    Niko1221
    on Oct 3, 2026
    Owner

    Thanks for the careful report and the clean repros. What happens: in this model's vocabulary </think> is one token (248069). The model writes that same token whether it means to end its thinking or to quote the tag. The server switches from reasoning to the answer at the first one, as other servers for Qwen models do, so a tag quoted inside the thinking ends the thinking early. In the empty-answer case the model most likely writes </think> while quoting it and then ends its turn right away, so no answer text comes.

    A </think> typed in a message is read as that same token today, which probably pushes the model into this. The plan: encode a literal <think> / </think> inside message text as plain text instead of the special token (as image markers in text already are), then measure whether the model quotes it as text. The server can't tell a tag the model writes mid-thought from a real end of thinking, so that part depends on the model. Keeping this open for that.

    @beatcollector: an output that ends inside a tool call is a separate case. A call that never closes is not reported as a tool call (#211), so a non-streamed reply can come back empty. Could you open it as its own issue with a repro on 0.1.38? Then we can look at returning that text as content.

  4. ampersandru commented on Oct 3, 2026

    @ampersandru
    ampersandru
    on Oct 3, 2026

    I’m seeing similar symptoms with Hermes Agent → LiteLLM → Strata, using Qwen3.8-Flash-Next IQ3_XXS. Hermes intermittently ends a task before completing it and displays the model’s reasoning as its final chat response.

    My setup:

    • Strata 0.1.37, according to Hermes’s diagnostic report.
    • RTX 3090 + RTX 5060 Ti, GPUS=all.
    • Context: 204800, KV: k8v4, vision enabled.
    • Hermes primarily used a LiteLLM alias with reasoning_effort: low and max_tokens: 32768.
    • Hermes has reasoning_echo: false and tool_use_enforcement: true.

    Hermes’s self-investigation reported repeated log entries:

    Reasoning-only clean stop (...) — returning the reasoning as the final response
(model=LLM:thinking-tools ...)

    It also reported that the corresponding stored assistant messages had:

    • Empty content.
    • No structured tool_calls.
    • Non-empty reasoning containing literal <tool_call> markup.

    This appears consistent with an intended tool action being returned as reasoning without a structured call, after which Hermes treats the response as the end of the task. I haven’t independently captured the raw Strata response yet, so I can’t confirm whether this is reasoning-boundary parsing, incomplete-call handling, or something introduced by LiteLLM.

    Small tool probes reportedly succeeded with both tool_choice: auto and required, in streaming and non-streaming modes. The recurring failures appeared during longer conversations with multiple tool turns.

    There were also separate rate-limit and fallback-connectivity errors, but those produced explicit errors rather than this reasoning-only ending.

    I’m testing thinking disabled, a smaller context allowance, and an explicit reasoning budget on the thinking route. I don’t yet have results confirming that these resolve the problem. I can follow up with a redacted request/response captured directly from Strata and through LiteLLM to help isolate the failing layer.

  5. sheldon-claw commented on Oct 4, 2026

    @sheldon-claw
    sheldon-claw
    on Oct 4, 2026

    Another data point for the "reasoning-only clean stop" that @ampersandru describes, with counts from a full agent session. In our case the model never writes </think> at all: the turn's reasoning ends with a complete, well-formed tool call, and the turn stops cleanly.

    Setup: Strata 0.1.38 (unmodified), Qwen3.8-Flash-Next IQ3_S, Windows 11, RTX 5090, context 262144, MTP on, vision on. Client: Cherry Studio 2.0.14's built-in agent (pi runtime) over /v1/chat/completions, streaming, 8 tools, max_tokens 29696. Every request resolves to thinking on.

    Counts (all Flash Next assistant turns in the agent's session files):

    turns
    assistant turns 318 (263 with a tool call)
    no content and no tool call 13 (12 stop, 1 a client-side error)
    ...reasoning ends with a complete <tool_call>\n<function=…>…</function>\n</tool_call> 7 (6 of them stop)
    ...reasoning ends with the user-facing answer 2
    ...reasoning stops right where the model is about to quote a tag (it was discussing this very bug) 4, which looks like the case this issue opened with

    None of the 13 contain </think>. None hit length; they used 61 to 2025 output tokens. The 7 reasoning-only turns end exactly like a normal call, for example:

    …
</parameter>
<parameter=timeout>
40
</parameter>
</function>
</tool_call>

    Since the parser stays in the reasoning state until </think>, the whole call goes out as reasoning_content. The client sees no content and no tool_calls, so it ends the task.

    For comparison: a Q4 GGUF of Qwen3.8-27B on llama.cpp in the same client and agent has 6280 turns with thinking, and 0 of them have tool-call markup in the reasoning. Its clean stop turns with no content and no tool call come to 15 out of 6367. That is a different server, so this compares setups, not just the two models.

  6. Niko1221
    added a commit that references this issue on Oct 4, 2026

    serve: a literal <think> / </think> in a message is encoded as text (#…

    e43d072

  7. Niko1221 commented on Oct 4, 2026

    @Niko1221
    Niko1221
    on Oct 4, 2026
    Owner

    Fixed in 0.1.39, on the input side: a literal <think> or </think> inside message text is now read by the model as plain text, not as its reasoning markers. That covers user messages, tool results, system prompts, earlier assistant turns, tool call arguments and tool descriptions. The tokenizer matches those two strings as special tokens wherever they appear (llama.cpp does the same), so before, your first prompt gave the model a real end-of-thinking token and the answer came back empty. The template's own markers are unchanged, and a prompt without the tags is encoded exactly as before. One thing stays a model limit: when the model itself writes </think> while quoting it in its reasoning, that still ends the reasoning (your second case).

  8. Niko1221
    closed this as completedon Oct 4, 2026
  9. sergqwer
    added a commit that references this issue on Oct 4, 2026

    serve: the vision markers inside a message's text stay text and no lo…

    1273345
  10. blange48
    added a commit that references this issue on Oct 4, 2026

    serve: the prompt is encoded from the last special it shares with a r…

    f3089a2
  11. gputier
    added a commit that references this issue on Oct 5, 2026

    serve: the prompt is tokenised from the last shared prefix, not from …

    bfe4bcf
  12. Niko1221
    added 3 commits that reference this issue on Oct 6, 2026

    serve: the vision markers inside a message's text stay text and no lo…

    2508e5a

    serve: a literal <think> / </think> in a message is encoded as text (#…

    c16f31f

    serve: the vision markers inside a message's text stay text and no lo…

    d6527e3
  13. TwoAnts
    added a commit that references this issue on Oct 6, 2026

    serve: a marker the reply quotes stays text, and a turn that ends mid…

    010d0d8

  14. 16 remaining items

  15. TwoAnts
    added a commit that references this issue on Oct 11, 2026

    serve: a marker the reply quotes stays text, and a turn that ends mid…

    f99e638
  16. 47Hunter47
    added 4 commits that reference this issue on Oct 12, 2026

    serve: the token that ended the turn is given back as text (Niko1221#…

    06dfdc8

    serve: the token that ended the turn is given back as text (Niko1221#…

    067aa7d

    serve: the token that ended the turn is given back as text (Niko1221#…

    306d156

    serve: the token that ended the turn is given back as text (Niko1221#…

    00027bd
  17. TwoAnts
    added a commit that references this issue on Oct 14, 2026

    serve: a marker the reply quotes stays text, and a turn that ends mid…

    3072913
  18. giridhart
    added a commit that references this issue on Oct 15, 2026

    serve: a </think> the reasoning quotes stays reasoning instead of end…

    bbd79e4
  19. TwoAnts
    added a commit that references this issue on Oct 17, 2026

    serve: a marker the reply quotes stays text, and a turn that ends mid…

    24041c9
  20. Rarebigshot17
    added 2 commits that reference this issue on Oct 18, 2026

    test_server region C: tidy the GLM port's wrapped lines and the Niko1…

    51bd533

    Reasoning tools tests: a quoted </think> ends the thinking under GLM

    efbd1db
  21. 47Hunter47
    added a commit that references this issue on Oct 19, 2026

    serve: the token that ended the turn is given back as text (Niko1221#…

    2c3c413
  22. DarthCY-K
    added a commit that references this issue on Oct 20, 2026

    serve: the vision markers inside a message's text stay text (Niko1221…

    dc31d8f
  23. TwoAnts
    added 3 commits that reference this issue on Oct 20, 2026

    serve: a marker the reply quotes stays text, and a turn that ends mid…

    92618c0

    serve: a marker the reply quotes stays text, and a turn that ends mid…

    9dc25ca

    serve: a marker the reply quotes stays text, and a turn that ends mid…

    58b8238
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions