Skip to content

fix(gateway): replay a Responses text + tool-call turn as one assistant message - #60

Open
ScottLiao920 wants to merge 1 commit into
NVIDIA-NeMo:stablefrom
ScottLiao920:fix/merge-assistant-text-with-tool-calls
Open

ScottLiao920 wants to merge 1 commit into
NVIDIA-NeMo:stablefrom
ScottLiao920:fix/merge-assistant-text-with-tool-calls

Conversation

@ScottLiao920

Copy link
Copy Markdown

Fixes #59.

What

_convert_input_items_to_messages now post-processes the converted messages: a text-only assistant message directly followed by an assistant message that carries only tool calls is merged into that message (reasoning_content of both is joined). Text-only assistant messages not followed by tool calls stay separate.

Why

See the issue: a model turn that has text followed by a tool call is stored in the Responses history as a message item plus a function_call item, and was replayed as two consecutive assistant chat messages.

Tests

Two tests added to tests/gateway/test_transform_openai_responses.py: the merge case, and separate text-only assistant turns staying separate.

Status: I could not run the repository's test suite in my environment; I checked the merge function in isolation (cases: merge, separate text-only turns, reasoning joined, tool-call message with content not merged, parallel calls merged together, idempotence). Please run the test file in CI before merging.


🤖 Generated with Claude Code

…nt message

A Responses history stores a model turn that has text followed by a tool call as a `message` item plus a
`function_call` item. _convert_input_items_to_messages emitted two consecutive assistant chat messages for it
(text only, then tool_calls only). Merge a text-only assistant message that is directly followed by an assistant
message carrying only tool calls into one message (reasoning_content of both is joined). Text-only assistant
messages not followed by tool calls stay separate. Adds two tests.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Liao Chang <35758766+ScottLiao920@users.noreply.github.com>
@ScottLiao920
ScottLiao920 force-pushed the fix/merge-assistant-text-with-tool-calls branch from 47866d2 to d5c8c33 Compare October 26, 2026 09:29
@jasonchen505

Copy link
Copy Markdown
Contributor

The merge only fires when the previous message has truthy content, but openai_responses.py lines 643 to 648 append an assistant message even when content is empty. A turn with an empty-text message item plus a function_call would still replay as two consecutive assistant messages after this fix. Do you see this case in practice with codex?

@ScottLiao920

Copy link
Copy Markdown
Author

Thanks for pointing this out. You're right: the current prev.get("content") guard leaves an explicit empty assistant message followed by a function call unmerged. I reproduced that with synthetic inputs ("", null, [], and an empty output_text item).

I did not observe that empty-message case in the real Codex run below: neither arm had a request containing an empty assistant message item. That is an observation from this sample, not a claim that Codex can never send one. This experiment validates the non-empty text + tool-call fix; it does not validate coverage of the empty-text edge case.

I have now also tested this with the actual Codex CLI and this repository's native Polar gateway/converter, rather than only the separate gateway mentioned in #59:

Metric Upstream This PR
ALFWorld tasks solved 0/10 4/10
Episodes ending early before the environment was done 10/10 1/10
Requests containing the split adjacent assistant messages 10 0

All upstream episodes stopped after one environment action. Text + tool-call replies occurred naturally in both arms; no commentary or synthetic history was injected. The PR's six unsuccessful episodes were three action-limit failures, two context-limit failures, and one early stop. All remain in the denominator; no episode was retried or excluded.

To distinguish the two cases: a tool-call-only Chat message with content: null is normal and is not an empty Responses message item. In one upstream trace, the model generated non-empty text about finding a pencil together with act({"action":"look"}). Replay split that into a text-only assistant message and a separate tool-call assistant message. On the next turn it said it would check inventory, but emitted no tool call; the CLI then ended while the environment was still unfinished. The original bug is therefore observable without any empty-text message item.

Setup: unmodified official Codex CLI 0.115.0, Qwen3.5-4B (revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a), real ALFWorld stdio MCP, and native polar.gateway.server from upstream 45feeffe9aa9e40a3d26b8addfcb22b3eb05d0d5. The two arms loaded the complete Responses converter from that commit versus PR commit d5c8c33013897dfabc6b4e144a32196fce033378. Capture wrappers and decoding controls were identical; no tool-name, namespace, history, or streaming rewrites were added. Temperature 0, thinking disabled, seed 20261030, 32,768-token context, 4,096 output-token budget, 50 actions, 600 seconds per episode. The same ten valid_seen tasks were fixed before inference (all six previous pilot tasks plus four deterministic hash-selected tasks); arm order alternated by task.

Version caveat: I first tried 0.125.0, but its MCP tools arrived as namespace definitions, which the current converter drops in both arms. I therefore used 0.115.0, which passed the real MCP round-trip check without a compatibility patch. These results do not establish behavior for 0.125.0/latest Codex.

All 20 outcomes and action counts were checked against the environment logs; CLI events/native sessions and raw request/response logs were also retained and audited. This is a small full-CLI/native-gateway pilot, not the full Polar Docker rollout orchestration or SWE-bench, so I would treat it as evidence that preserving the original message structure matters here—not a general benchmark-gain claim.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Responses→Chat conversion replays a text + tool-call turn as two consecutive assistant messages

2 participants