Repository navigation
serve: preserve literal </think> across reasoning parsing - #1886
CC-David-CC wants to merge 9 commits into
Conversation
…ued (Niko1221#1814) The model is about to write its closing thinking tag and the turn-end token comes instead: finish_reason is "stop", the reasoning stops mid-sentence, and the client gets an empty reply. Reproduced on an RTX 3090 with Qwen3.8-Flash-Next IQ3_XXS on 0.1.41: 6 of 9 trigger runs (streaming and whole, with and without STRATA_PREFILL_CPU_SHARE). Five raw turns dumped with STRATA_DEBUG=1 are in serve/fixtures/literal_think_specimens.json: in each of them the text ends right where the tag would begin (or on a backtick) and the end-of-turn token follows. The parser cannot tell this end from a normal one - the tag is eight ordinary text tokens in this tokenizer, and a turn that really ends looks the same - and nothing in the token stream says "I meant it". So, like Niko1221#1053 does for the reasoning side, the reply is continued with the thinking closed the way Niko1221#123's wrap-up closes it (REASONING_CLOSE): the next pass's prompt is this one plus what was generated plus that close, so the engine continues from the prefix it already holds and the model answers. parser.state == "content" is the same end seen from the other side (the model wrote the tag as text, the parser took it as the marker and nothing followed); there the tag goes back into the reasoning text first. Opt-in, like Niko1221#1053: "literal_think_guard": true in strata-<model>.json, or STRATA_LITERAL_THINK_GUARD=1. Capped per reply (LITERAL_THINK_RETRIES = 2), so a model that keeps doing it cannot loop. serve/test_literal_think_guard.py replays the five real turns through a mock engine (no answer -> answered; answered and max-token-cut turns -> untouched; the cap). serve: 604 tests, the only failure is the pre-existing flaky telemetry thread count in test_responses.py (OK/FAILED/FAILED across three runs of it alone).
…rd (Niko1221#1814) Detection and continuation are separate things, and only the second one changes what a reply contains. The end - finish_reason "stop" inside the thinking, with no answer - is now always reported: a log line, totals.ended_inside_thinking (and /metrics' totals_ended_inside_thinking_total), and "ended_inside_thinking": true beside `choices` in the response (and in the stream's last chunk). Reporting needs no switch because it changes no output: a client that is told why its reply is empty can retry, or turn the guard on. The continuation stays opt-in ("literal_think_guard"), as in the first commit: it appends the close to the prompt and spends another pass. serve/test_literal_think_guard.py: the guard-off case still reports the end (and stays empty), the reply the guard answered is reported too, and a normal reply carries no flag. serve: 607 tests, the only failure is the pre-existing flaky telemetry thread count in test_responses.py.
… it (Niko1221#1814) The guard reopened the parser to reasoning in an earlier draft of this branch, and that call was lost when the block was rewritten: with parser.state == "content" - the model wrote the tag as ordinary text, the parser took it for the marker, and nothing followed - the appended close then landed in the ANSWER instead of closing the thinking. The tag belongs to the reasoning (the client still gets everything the model wrote), so parser.reopen_reasoning() goes back before the close is fed. serve/test_literal_think_guard.py replays that case too (ContentFlipEngine): the answer is the answer alone, the tag is not in it, and the tag is in the reasoning. The test fails without the call and passes with it.
|
if you dont want this i will maintain this for my own internal use |
|
Two things that may be useful for this branch.
For reference, the side of #1840 that is in your branch stays opt-in: the continuation and the reporting are both behind |
|
i have some heavy tests running: and more scheduled after this to confirm some things and make sure my Context Cow is stable so that i can abuse it tremendously then you'll not see anything major from me for like 3 weeks as i transition to dogfooding it with my own model internally. |
|
Thanks for the pull request. We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected. To reopen it takes about two minutes:
The rules are in CONTRIBUTING.md. |
Summary
Builds on the end-of-thinking recovery and captured real-world specimens from #1840, then makes parser transitions depend on the actual special-token identity. This keeps an ordinary
</think>string in reasoning when it is quoted or appears mid-line, including when the model continues reasoning afterward.Reproduction and fix
With #1840's end-of-turn recovery alone, a deterministic token-stream test puts a literal
</think>in ordinary prose and then continues reasoning. The parser switches to answer mode at the string; the remaining reasoning and quoted tag are routed to the answer. A second test emits the special close token inside an unclosed inline-code span and stops. The recovery continuation preserves the backtick boundary, but the close token is exposed as a structural close instead of quoted reasoning text.This branch adds token-aware parser signaling and bounded continuation handling. In the same streams, the full prose and quoted tag stay in reasoning, and the answer remains
42. The original #1840 empty-reasoning / no-answer recovery and its real-turn specimens remain included.Validation
main: 735 tests passed, 12 skipped.git diff --check upstream/main...HEAD: clean.These are deterministic service/parser tests; no live Qwen3.8 / dual-3090 rerun is claimed. The recovery/continuation guard remains opt-in, while token-identity parsing is always used.
Supersedes the standalone draft in #1839 and combines its parser-ambiguity fix with #1840's reproduction and recovery.