Skip to content

Recover empty and text-only tool responses in the LLM compatibility proxy - #120

Open
JeffPengCoder wants to merge 2 commits into
NVIDIA:mainfrom
JeffPengCoder:fix/tool-call-text-recovery-20261119
Open

JeffPengCoder wants to merge 2 commits into
NVIDIA:mainfrom
JeffPengCoder:fix/tool-call-text-recovery-20261119

Conversation

@JeffPengCoder

Copy link
Copy Markdown
Contributor

When a tool-capable model returns only whitespace or prints <run_kubectl args=[... ]} as ordinary text, the runtime receives no tool call. In a native messaging channel the run can finish without a visible reply, or show a command that never executed. This adds an opt-in recovery guard to the existing LLM compatibility proxy.

Set llm.requestCompatibility.toolCallTextRecovery.enabled=true to buffer single-choice, automatic-tool completions. If a completed response has no structured calls and is empty or contains recognizable invocation markup for a registered tool, the proxy retries once with the original conversation and tool definitions plus a corrective system instruction. It never converts the text into executable arguments or executes tools. Existing native calls pass through unchanged; runtime validation and authorization still apply. A second invalid response fails explicitly.

The setting defaults to false. Enabling it delays eligible streaming responses until generation completes and may consume one additional model request. Buffering is capped at 16 MiB, with a 180-second deadline across both requests. Named/disabled tool choice, multiple choices, ordinary prose, quoted/fenced examples and explicit silent-response markers retain their existing behavior. Detection covers recognizable invocation markup, not every malformed output or the factual quality of an answer.

Validation:

  • Full Node CI suite: 276 tests, 274 passed, 2 existing conditional skips, 0 failures. The 22 recovery tests cover JSON/SSE, empty replies, native-call pass-through, bounded retries, compressed responses, incomplete streams, size limits, deadlines, and non-execution of malformed arguments.
  • Standalone inference tests, both Helm lints, documentation bundle, strict Sphinx build, SPDX checks and git diff --check passed. The unchanged upstream BCM server suite passes all 24 tests on Linux; its process-group termination test hits macOS PermissionError, independently of this change.
  • A live Qwen3.6 deployment reproduced an empty first response and recovered a native tool call. The runtime subsequently returned actual read-only Kubernetes tool results and a final answer. A new human Feishu group message also produced native calls and an automatic report, confirmed by the recipient. Some query arguments still failed validation/data access; this verifies protocol recovery and delivery, not complete diagnostic accuracy.

The source branch is based on main. Changes are limited to the compatibility guard, its chart wiring, tests and documentation; existing channel configuration is unchanged. The commit is DCO signed off.

Signed-off-by: Jeff Peng <jepeng@nvidia.com>
Signed-off-by: Jeff Peng <jepeng@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant