No valid checkpoint to restore + Max length reached on multi-turn tool calls (Qwen3.6-MoE, NPU) #744

Activity

1. winaes13-ux commented on Sep 18, 2026

   

   winaes13-ux

   on Sep 18, 2026

   Author

   More actions

   Addendum: default_context_length sizing for qwen3.6-moe

   Context

   The patch described above fixes the restore() failure path, but it does not remove the need for a correctly sized default_context_length. Without a sufficiently large MAX_L, the same multi-turn session still fails later with Max length reached once the prompt has grown enough.

   Why default_context_length matters

   When restore() returns a negative value, the patched Qwen3_6_MOE::insert() resets total_tokens to 0 and clears token_history / checkpoint_his. _shared_insert() then performs a full prefill of the entire prompt:

   total_tokens (0) + tokens.size() (full prompt) >= MAX_L ?
   

   If MAX_L is smaller than the prompt size, the prefill is rejected:

   [FLM]  No valid checkpoint to restore
   [WARNING]  Max length reached, stopping prefilling...
   [FLM]  Creating checkpoint at context length 0
   

   and the server returns HTTP 400:

   {"error":{"message":"Max length reached!","type":"model_error","code":400}}

   MAX_L is read from model_list.json → default_context_length. For qwen3.6-moe:35b-a3b the upstream value is 32 768, which is not enough for long agent sessions.

   Sizing rule

   A full re-prefill happens on top of whatever was already in the KV cache before the reset, so MAX_L must cover the worst-case prompt with a 2× headroom:

   worst_case_prompt = system_prompt
                     + tools_definition
                     + conversation_history
                     + max_tokens
   
   default_context_length = 2 * worst_case_prompt
   

   Component sizes (OpenCode, OpenAI-compatible client)

   Component	Typical size	Notes
   system_prompt	~2 000 tokens	OpenCode system prompt + AGENTS.md + MCP hints
   tools_definition	~6 000 tokens	20 MCP tools: codebase-memory-mcp (15) + context7 (2) + built-ins
   conversation_history	~16 000 tokens	typical multi-turn agent session
   max_tokens	4 096 tokens	request-level max_tokens
   worst_case_prompt	~28 096 tokens	sum of the above
   default_context_length	65 536	2 × 28 096 = 56 192 → rounded up

   Rounding table

   worst_case_prompt	2 × worst_case_prompt	default_context_length (rounded)
   ≤ 16 384	≤ 32 768	32 768
   ≤ 32 768	≤ 65 536	65 536 ← current OpenCode configuration
   ≤ 65 536	≤ 131 072	131 072
   ≤ 131 072	≤ 262 144	262 144 (native Qwen3.6-MoE maximum)
   > 131 072	> 262 144	262 144 (clamped to model maximum)

   Workaround used locally

   Until the model-side fix lands, the user-side copy of model_list.json is patched via FLM_CONFIG_PATH:

   export FLM_CONFIG_PATH="$HOME/.config/flm/model_list.json"
   export FLM_XCLBIN_PATH="/opt/fastflowlm/share/flm"
   
   mkdir -p "$(dirname "$FLM_CONFIG_PATH")"
   cp /opt/fastflowlm/share/flm/model_list.json "$FLM_CONFIG_PATH"
   
   # raise default_context_length for qwen3.6-moe to 65536
   sed -i 's/"default_context_length": 32768/"default_context_length": 65536/g' \
       "$FLM_CONFIG_PATH"

   FLM_CONFIG_PATH is read by common/utils.cpp (get_models_directory() / get_model_path()), and the user copy survives FLM upgrades, unlike the system copy under /opt/fastflowlm/share/flm/.

   Verification

   With the patch and default_context_length = 65 536 in place, a 28-message agent session completes without Max length reached. The longest observed prefill in the test session was:

   [FLM]  Prefill chunk 1/8 with 4096 tokens
   ...
   [FLM]  Prefill chunk 8/8 with 2178 tokens
   [FLM]  Creating checkpoint at context length 30850
   [FLM]  Start generating...
   

   30 850 < 32 768 on the first attempt, but the failure path adds another full prefill on top of it — which is why MAX_L must be sized for 2 × worst_case_prompt, not 1 × worst_case_prompt.

   Suggested fix in the repository

   1. Apply the insert() patch from the main report.
   2. Either raise default_context_length for qwen3.6-moe:35b-a3b in model_list.json to a value ≥ 2 × worst_case_prompt for the intended client configuration, or document the sizing rule so users can adjust it themselves.
   3. Optionally, expose default_context_length as a runtime override (e.g. --ctx-len) that is independent of the model registry, so users don’t need to edit model_list.json at all.

   ---
2. winaes13-ux commented on Sep 21, 2026

   

   winaes13-ux

   on Sep 21, 2026

   Last edited by winaes13-ux

   Author

   More actions

   Addendum 2: Official chat_template.jinja for Qwen3.6-MoE breaks tool calling on C++ engines

   Summary

   The official chat_template.jinja shipped with Qwen3.6-35B-A3B-NPU2 uses Python-specific Jinja constructs that are not supported by the C++ minja runtime inside FastFlowLM. As a result, tool calling is unreliable: the template either fails to render, drops tool arguments, or produces malformed XML that the parser cannot recover from. On long agent sessions this manifests as Max length reached and HTTP 400 responses, even when the context is well below default_context_length.

   Affected component

   • Model: qwen3.6-moe:35b-a3b (NPU2)
   • File: ~/.config/flm/models/Qwen3.6-35B-A3B-NPU2/chat_template.jinja
   • Runtime: FastFlowLM’s C++ Jinja engine (minja)

   Root cause

   The upstream Qwen3.6 template relies on Python-only Jinja filters, most notably:

   • |items — used to iterate over tool.arguments and tool.parameters.properties
   • |safe — used to mark JSON-serialized fragments as safe
   • |string on non-string values in a few branches

   FastFlowLM’s minja runtime does not implement these filters. Depending on the code path, this leads to one of three failure modes:

   1. Template render exception — the whole request is rejected.
   2. Silent argument drop — the tool call is emitted without arguments, so the agent receives an empty payload.
   3. Malformed XML — the parser receives a partially rendered <tool_call> block and cannot recover, which in turn triggers the No valid checkpoint to restore path and eventually Max length reached.

   All three failure modes are reproducible on a stock FastFlowLM 1.0.5 installation without any user modifications.

   Reproduction

   1. Start flm serve qwen3.6-moe:35b-a3b.
   2. Send a request with a non-empty tools array (e.g. via OpenCode, Claude Code, or a direct curl).
   3. The first tool call may succeed; the follow-up turn (with role: "tool") frequently fails with Max length reached or with a malformed tool call.

   Evidence from the log

   [FLM]  Use cached prompt!
   [FLM]  Matched 2 out of 4 messages (2 new to prefill).
   [FLM]  Start prefill...
   [FLM]  Total images: 0
   [FLM]  No valid checkpoint to restore
   [WARNING]  Max length reached, stopping prefilling...
   [FLM]  Creating checkpoint at context length 0
   

   The failure is not caused by the context size itself — the same session completes when the template is replaced.

   Workaround

   Replace the official chat_template.jinja in the model directory with the community-maintained fixed template from:

   https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
   

   The fixed template (v22.5 at the time of writing) rewrites all Python-only filters into minijinja-safe equivalents and fixes several related issues, including:

   • Python-specific Jinja filters (|items, |safe) → universal constructs
   • Unclosed <think> blocks before tool calls
   • Empty {} tool arguments on multi-step turns
   • Duplicate <think></think> blocks in chat history
   • KV cache invalidation from mutated past turns

   Local patch applied

   ~/.config/flm/models/Qwen3.6-35B-A3B-NPU2/chat_template.jinja         # 28 234 bytes (fixed, v22.5)
   ~/.config/flm/models/Qwen3.6-35B-A3B-NPU2/chat_template.jinja.bak     #  7 764 bytes (official)
   

   The fixed template is roughly 3.6× larger than the official one because it contains the full reasoning / tool-call / error-recovery logic in a form the C++ runtime can execute.

   Persistence across upgrades

   The fixed template lives inside the model directory, not in the system registry. This means:

   • It survives FLM upgrades (the system model_list.json is never touched).
   • It is reset by flm pull --force, because the pull operation re-extracts the model directory from the Hugging Face repository.

   A post-pull check is therefore recommended:

   grep -c "froggeric" ~/.config/flm/models/Qwen3.6-35B-A3B-NPU2/chat_template.jinja

   If the output is 0, the official template has been restored and must be replaced again before tool calling will work reliably.

   Impact

   Any client that relies on multi-step tool calls (OpenCode, Claude Code, Cline, Cursor, and any OpenAI-compatible harness) is affected. Single-turn chat still works with the official template, but agentic workflows do not.

   Suggested fix in the repository

   1. Ship the minijinja-safe template as the default chat_template.jinja for qwen3.6-moe:35b-a3b (and any other Qwen 3.5/3.6/3.8 variants that use the same Python-only filters).
   2. Add a template-compatibility check at model load time that logs a warning when the template contains filters unsupported by minja.
   3. Document the flm pull --force behaviour so users know the template will be overwritten on re-pull.