Skip to content

tests: hrx_long_prompt turns thinking off through chat_template_kwargs - #399

Merged
bong-water-water-bong merged 2 commits into
mainfrom
fix/hrx-long-prompt-thinking
Oct 16, 2026
Merged

bong-water-water-bong merged 2 commits into
mainfrom
fix/hrx-long-prompt-thinking

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

tests/hrx_long_prompt.py sends enable_thinking: false at the top level of the request. llama-server ignores that field, so Qwen3 spends the probe's 16 tokens thinking (finish=length) and the 7K-token prompt returns '' on every pin. The probe can't serve as a gate in that state.

This keeps the top-level field and adds chat_template_kwargs: {"enable_thinking": false}, which llama-server honours, in both requests.

Evidence from the PR #393 gate on strixhalo, 2026-10-12, Qwen3-0.6B on HRX0:

  • Unmodified probe: FAIL on the PR build and on main pins (0e255d77 / 98d05d94).
  • Same prompt with the template kwargs: '7341' at 7094 tokens on both builds.
  • Server started with --chat-template-kwargs '{"enable_thinking":false}': the probe passes 3/3 on both.

tests/hrx_long_prompt_test.py passes.

🤖 Generated with Claude Code

llama-server ignores a top-level enable_thinking, so Qwen3 spent the probe's
16 tokens thinking and the 7K prompt returned '' on every pin. Keep the
top-level field for servers that read it and add the chat_template_kwargs
form llama-server honours. Measured on strixhalo during the PR #393 gate:
the unmodified probe fails on both the PR and the main pins; with thinking
off through the template kwargs it passes 3/3 on both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 62ff16a)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

393 - Partially compliant

Compliant requirements:

  • Bump HRX submodules to llama.cpp 2bdb04a0928c and hrx-system 98d05d94a9f9
  • Run long-prompt correctness probe and test-backend-ops -b HRX0 on Strix Halo
  • Record the exact validated hrx-system pin in config/hrx-long-prompt-validated-pin.config
  • Validate that the probe passes with the updated configuration

Non-compliant requirements:

  • Ensure CI builds without HRX (this requirement is not directly verifiable from the diff, but implied by the PR description)

Requires further human verification:

  • CI build verification (not directly visible in diff)
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Inconsistent enable_thinking handling

The test adds chat_template_kwargs: {"enable_thinking": false} to override the top-level enable_thinking: False field, which suggests that the top-level field is not being respected by the server. This could indicate a potential inconsistency or misconfiguration in how the server handles the enable_thinking parameter, which should ideally be consistently processed at the top level.

"enable_thinking": False,
"chat_template_kwargs": {"enable_thinking": False},
Inconsistent enable_thinking handling

The test adds chat_template_kwargs: {"enable_thinking": false} to override the top-level enable_thinking: False field, which suggests that the top-level field is not being respected by the server. This could indicate a potential inconsistency or misconfiguration in how the server handles the enable_thinking parameter, which should ideally be consistently processed at the top level.

"enable_thinking": False,
"chat_template_kwargs": {"enable_thinking": False},

@github-actions

Copy link
Copy Markdown

Persistent review updated to latest commit 62ff16a

@bong-water-water-bong
bong-water-water-bong merged commit 7a7361a into main Oct 16, 2026
7 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the fix/hrx-long-prompt-thinking branch October 16, 2026 03:49
bong-water-water-bong pushed a commit that referenced this pull request Oct 16, 2026
Both sides apply the same thinking-off fix (chat_template_kwargs enable_thinking
False); #399 added it on main, this branch added it too. Keep this branch's
explanatory comment; the test code is otherwise identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant