common/parser: add proper reasoning tag prefill reading by pwilkin · Pull Request #20424 · ggml-org/llama.cpp

pwilkin · 2026-03-11T20:49:33Z

This changes the erroneous behavior of the autoparser that ascribed thinking behavior to templates. As people rightly mentioned, some models have dynamic or hybrid reasoning - they can reason or not depending on some switches and even the template behavior can change due to this (i.e. inserting <think> in assistant prefill after a "no_think" appears in a user message).

Therefore, the FORCED_OPEN and FORCED_CLOSED formats are gone. The parser will now just detect models with tagged reasoning, i.e. an opening and closing reasoning marker (deleted DELIMITER also since it's a special case with the opening marker being empty). However, it will check the assistant prefill for those markers and will append them to the input for the grammar and the parser, so that they are taken into account, therefore just simplifying the parsing mechanism since it doesn't now have to differentiate whether the <think>' / ` was added by the template or generated by the model.

pwilkin · 2026-03-11T20:51:47Z

Fixes #20356
Fixes #20325
Fixes #20265

This also clears the ground for disabling grammar triggers inside reasoning loops in a subsequent PR, which would resolve #20260

aldehir · 2026-03-11T21:17:53Z

Dumb question, why not find the start of the assistant message and prepend that?

I agree it would be easier to parse if we had a "prefill" of some sort that normalizes the input, such that we can handle the logic in the grammar and not through flags. However, if we're going this route I would look into prepending the start of the entire assistant message. This will also open the door for parsing output from requests with an assistant prefill.

pwilkin · 2026-03-11T21:30:35Z

Yeah, that would be the logical conclusion, but for now it's easier for me just to extract the reasoning markers since finding the actual start of the assistant message is nontrivial.

aldehir · 2026-03-11T23:09:26Z

Qwen3.5 uses

<think>\n\n</think>\n\n

{%- if enable_thinking is defined and enable_thinking is false %}
{{- '<think>\n\n</think>\n\n' }}
{%- else %}
{{- '<think>\n' }}
{%- endif %}

however,

      "reasoning_prefill": "<think></think>\n\n",

It probably doesn't matter for this model, but it is technically not adhering to the template.

aldehir · 2026-03-11T23:15:21Z

    {
      "id": 248045,
      "piece": "<|im_start|>"
    },
    {
      "id": 74455,
      "piece": "assistant"
    },
    {
      "id": 198,
      "piece": "\n"
    },
    {
      "id": 248068,
      "piece": "<think>"
    },
    {
      "id": 271,
      "piece": "\n\n"
    },
    {
      "id": 248069,
      "piece": "</think>"
    },
    {
      "id": 271,
      "piece": "\n\n"
    }

Maybe set reasoning_prefill from the start of the opening tag to the end of the prompt?

aldehir · 2026-03-11T23:32:03Z

finding the actual start of the assistant message is nontrivial.

Run the template once with add_generation_prompt = false, capture the size, run again with true, extract the string content that spans the delta? I think that would work in most cases.

pwilkin · 2026-03-12T00:27:07Z

That usually works, yeah 😀 I can try that and see what the results are (this is what calculate_diff_split from the analyzer does BTW). I'm just worried about some weird edge cases.

bsdice · 2026-03-14T11:10:37Z

Nice patch! With model https://huggingface.co/mradermacher/Qwen3.5-40B-Claude-4.5-Opus-High-Reasoning-Thinking-GGUF the patches fix webui getting confused on /think and not splitting correctly reasoning and generation part. Build llama.cpp-cuda-git-b8334.r9.710878a7dd-1.

pwilkin · 2026-03-14T14:50:50Z

@aldehir changed the prefill extraction behavior to the differential one you mentioned.

aldehir · 2026-03-14T19:06:06Z

@@ -211,7 +211,7 @@ struct common_chat_params {
    std::string                         prompt;
    std::string                         grammar;
    bool                                grammar_lazy         = false;
-    bool                                thinking_forced_open = false;
+    std::string                         prefill;


Think we name this generation_prompt? It lines up with the add_generation_prompt flag.

aldehir · 2026-03-14T20:39:17Z

+    bool clear_reasoning_start = false;
+    if (inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE &&
+        autoparser.reasoning.mode != reasoning_mode::NONE &&
+        !autoparser.reasoning.end.empty()) {
+        const auto & r_start    = autoparser.reasoning.start;
+        const auto & r_end      = autoparser.reasoning.end;
+        auto         r_end_t    = trim_trailing_whitespace(r_end);
+        auto         r_start_t  = trim_trailing_whitespace(r_start);
+
+        if (!r_start_t.empty()) {
+            auto start_pos = prompt_to_search.rfind(r_start_t);
+            if (start_pos != std::string::npos) {
+                std::string from_start = prompt_to_search.substr(start_pos);
+                auto         fs_trimmed = trim_trailing_whitespace(from_start);
+
+                if (string_ends_with(fs_trimmed, r_end_t)) {
+                    data.prefill = r_start + r_end;
+                } else if (string_ends_with(fs_trimmed, r_start_t)) {
+                    data.prefill = from_start;
+                } else {
+                    clear_reasoning_start = true;
+                }
+            }
+        }
+    }


So my understanding is: we have a generation prompt G, we can create a parser that accepts G[0:max(G.size(), G.index_of(reasoning_start))] + (reasoning_start + reasoning + reasoning end)? + .... Then we can do away with all the trim logic.

The benefit is that now the parser can properly parse assistant prefill from the user, since the parser starts from the beginning of the assistant message.

I see that Mistral's templates have no generation prompt, so G = "". But this is fine, because the model emits the [THINK] tag. So the above still works.

This workaround is mostly for Apriel that has a delimited thinking format and inserts a header "Thinking chain starts here: " or something like that as the generation prompt which acts as a quasi-reasoning marker that we want to strip.

pwilkin · 2026-03-15T01:40:25Z

@aldehir okay, that rewrite ended up being a bit bigger than I expected... but it's exactly the algorithm you mentioned now.

aldehir · 2026-03-15T01:59:19Z

Oh jeez, well it's <100 net LOC. I'll give it a whirl.

pwilkin · 2026-03-15T15:03:13Z

@aldehir happy to report I added another nice piece of code to make it work correctly with grammars / schemas :)

…llama.cpp/pull/20424)

* Implement proper prefill extraction * Refactor cli parameters, update docs, move reasoning budget sampler part to common/reasoning-budget.cpp * Update tools/server/server-task.cpp * refactor: move grammars to variant, remove grammar_external, handle exception internally * Make code less C++y Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

…regression) (ggml-org#20777) * chat : fix out_of_range crash in throw path (ggml-org#20424 regression) ggml-org#20424 introduced effective_input = generation_prompt + input, but the throw path uses input.substr(result.end) where result.end is a position within effective_input. Every thinking model with a non-empty generation_prompt crashes with std::out_of_range instead of the intended error message. Test crashes on unpatched master, passes with fix: cmake -B build -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_TOOLS=OFF cmake --build build --target test-chat ./build/bin/test-chat * Update test-chat.cpp * Update test-chat.cpp * Update test-chat.cpp --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

* Implement proper prefill extraction * Refactor cli parameters, update docs, move reasoning budget sampler part to common/reasoning-budget.cpp * Update tools/server/server-task.cpp * refactor: move grammars to variant, remove grammar_external, handle exception internally * Make code less C++y Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

…regression) (ggml-org#20777) * chat : fix out_of_range crash in throw path (ggml-org#20424 regression) ggml-org#20424 introduced effective_input = generation_prompt + input, but the throw path uses input.substr(result.end) where result.end is a position within effective_input. Every thinking model with a non-empty generation_prompt crashes with std::out_of_range instead of the intended error message. Test crashes on unpatched master, passes with fix: cmake -B build -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_TOOLS=OFF cmake --build build --target test-chat ./build/bin/test-chat * Update test-chat.cpp * Update test-chat.cpp * Update test-chat.cpp --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

xhinker · 2026-05-19T05:24:55Z

This PR actually introduce a NEW BUG!
now set reasoning_format none, will lead to:

during streaming, no tag output, while \n\n output, it break a lot logic
during non-streaming, everything is ok

This is A BUG

aldehir · 2026-05-19T05:36:57Z

This PR actually introduce a NEW BUG!

now set reasoning_format none, will lead to:

during streaming, no tag output, while \n\n output, it break a lot logic

during non-streaming, everything is ok

This is A BUG

Create a new issue and fill out all the information in the template. Thank you.

xhinker · 2026-05-19T06:29:46Z

This PR actually introduce a NEW BUG!
now set reasoning_format none, will lead to:

during streaming, no tag output, while \n\n output, it break a lot logic

during non-streaming, everything is ok

This is A BUG

Create a new issue and fill out all the information in the template. Thank you.

here: #23320

* Implement proper prefill extraction * Refactor cli parameters, update docs, move reasoning budget sampler part to common/reasoning-budget.cpp * Update tools/server/server-task.cpp * refactor: move grammars to variant, remove grammar_external, handle exception internally * Make code less C++y Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

…regression) (ggml-org#20777) * chat : fix out_of_range crash in throw path (ggml-org#20424 regression) ggml-org#20424 introduced effective_input = generation_prompt + input, but the throw path uses input.substr(result.end) where result.end is a position within effective_input. Every thinking model with a non-empty generation_prompt crashes with std::out_of_range instead of the intended error message. Test crashes on unpatched master, passes with fix: cmake -B build -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_TOOLS=OFF cmake --build build --target test-chat ./build/bin/test-chat * Update test-chat.cpp * Update test-chat.cpp * Update test-chat.cpp --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

* Implement proper prefill extraction * Refactor cli parameters, update docs, move reasoning budget sampler part to common/reasoning-budget.cpp * Update tools/server/server-task.cpp * refactor: move grammars to variant, remove grammar_external, handle exception internally * Make code less C++y Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

…on) (#20777) * chat : fix out_of_range crash in throw path (#20424 regression) #20424 introduced effective_input = generation_prompt + input, but the throw path uses input.substr(result.end) where result.end is a position within effective_input. Every thinking model with a non-empty generation_prompt crashes with std::out_of_range instead of the intended error message. Test crashes on unpatched master, passes with fix: cmake -B build -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_TOOLS=OFF cmake --build build --target test-chat ./build/bin/test-chat * Update test-chat.cpp * Update test-chat.cpp * Update test-chat.cpp --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

* Implement proper prefill extraction * Refactor cli parameters, update docs, move reasoning budget sampler part to common/reasoning-budget.cpp * Update tools/server/server-task.cpp * refactor: move grammars to variant, remove grammar_external, handle exception internally * Make code less C++y Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

…regression) (ggml-org#20777) * chat : fix out_of_range crash in throw path (ggml-org#20424 regression) ggml-org#20424 introduced effective_input = generation_prompt + input, but the throw path uses input.substr(result.end) where result.end is a position within effective_input. Every thinking model with a non-empty generation_prompt crashes with std::out_of_range instead of the intended error message. Test crashes on unpatched master, passes with fix: cmake -B build -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_TOOLS=OFF cmake --build build --target test-chat ./build/bin/test-chat * Update test-chat.cpp * Update test-chat.cpp * Update test-chat.cpp --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

…on prompt The fallback split `effective_input`, which since ggml-org#20424 carries params.generation_prompt in front of the model's output. On a thinking template that prompt ("<|im_start|>assistant\n<think>\n" on Qwen3.5) does not start with "<think>", so the leading-tag strip never matched and the whole prompt was copied into reasoning_content: a 19-byte reasoning span came back as 56 bytes with the assistant turn header in it. Split `input` instead -- the generation prompt is not something the model produced. The test-chat case that asserts the strict throw now also runs the default path and checks the exact reasoning/content pair, with a non-empty generation_prompt in play, so a leak of this kind fails the test rather than the user. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

pwilkin requested review from aldehir, allozaur, ggerganov and ngxson as code owners March 11, 2026 20:49

github-actions Bot added documentation Improvements or additions to documentation testing Everything test related examples server labels Mar 11, 2026

pwilkin mentioned this pull request Mar 11, 2026

Eval bug: unsloth/Qwen3.5-35B-A3B-GGUF peg-native chat format parser fails when model outputs text before <tool_call> (thinking model + tool calling) #20260

Closed

loci-dev mentioned this pull request Mar 12, 2026

UPSTREAM PR #20424: common/parser: add proper reasoning tag prefill reading auroralabs-loci/llama.cpp#1247

Open

aldehir mentioned this pull request Mar 13, 2026

Eval bug: Response always starts with </think> tag when running Qwen3.5 9B #20516

Closed

pwilkin force-pushed the reasoning-prefill branch from 3bfb08f to 4083259 Compare March 14, 2026 14:49

aldehir reviewed Mar 14, 2026

View reviewed changes

pwilkin requested review from a team as code owners March 15, 2026 15:02

pwilkin mentioned this pull request Mar 15, 2026

Handle reasoning budget #20297

Merged

EZForever mentioned this pull request Mar 24, 2026

Eval bug: Autoparser misplaces non-thinking content with NVIDIA-Nemotron-Nano-9B-v2 #20325

Closed

gucasbrg mentioned this pull request Mar 29, 2026

[Bug]: Qwen3.5-27B tool call parsing still broken after PR #20424 (build 8576) #21158

Closed

ysimonson added a commit to ysimonson/llama-cpp-rs that referenced this pull request Apr 5, 2026

Remove thinking_forced_open (this was removed in github.com/ggml-org/…

ab68554

…llama.cpp/pull/20424)

ysimonson mentioned this pull request Apr 5, 2026

Sync to latest llama.cpp utilityai/llama-cpp-rs#983

Merged

joshvoigts mentioned this pull request Apr 6, 2026

Misc. bug: Chat template thinking block head <think> tag is not rendered / generated correctly (MiniMax-M2.5 + possibly other models) #21465

Closed

aldehir mentioned this pull request Apr 14, 2026

common : add common_chat_split_by_role #21885

Closed

ServeurpersoCom mentioned this pull request May 5, 2026

server, webui: support continue generation on reasoning models #22727

Merged

toyfer mentioned this pull request Jul 28, 2026

[B] llama.cpp 側で thinking を完全抑制する起動オプションを調査・導入 toyfer/AITranscribe-Electron#63

Closed

4 tasks

Conversation

pwilkin commented Mar 11, 2026

Uh oh!

pwilkin commented Mar 11, 2026

Uh oh!

aldehir commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwilkin commented Mar 11, 2026

Uh oh!

aldehir commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aldehir commented Mar 11, 2026

Uh oh!

aldehir commented Mar 11, 2026

Uh oh!

pwilkin commented Mar 12, 2026

Uh oh!

bsdice commented Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwilkin commented Mar 14, 2026

Uh oh!

aldehir Mar 14, 2026

Choose a reason for hiding this comment

Uh oh!

aldehir Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pwilkin Mar 14, 2026

Choose a reason for hiding this comment

Uh oh!

pwilkin commented Mar 15, 2026

Uh oh!

aldehir commented Mar 15, 2026

Uh oh!

pwilkin commented Mar 15, 2026

Uh oh!

xhinker commented May 19, 2026

Uh oh!

aldehir commented May 19, 2026

Uh oh!

xhinker commented May 19, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

aldehir commented Mar 11, 2026 •

edited

Loading

aldehir commented Mar 11, 2026 •

edited

Loading

bsdice commented Mar 14, 2026 •

edited

Loading

aldehir Mar 14, 2026 •

edited

Loading