Skip to content

fix(models): stream LiteLlm function-call args instead of buffering - #7040

Open
claxman wants to merge 2 commits into
google:mainfrom
claxman:fix/5342-litellm-partial-function-chunks
Open

fix(models): stream LiteLlm function-call args instead of buffering#7040
claxman wants to merge 2 commits into
google:mainfrom
claxman:fix/5342-litellm-partial-function-chunks

Conversation

@claxman

@claxman claxman commented Sep 7, 2026

Copy link
Copy Markdown

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
LiteLlm's FunctionChunk branch appends to args_parts and only yields on finish_reason. On b0180620, test_streaming_tool_call_args_assembled_from_many_fragments passed with len(responses) == 1, and grep PROGRESSIVE_SSE_STREAMING src/google/adk/models/lite_llm.py was empty. Long tool-call args do not stream.

Solution:
When PROGRESSIVE_SSE_STREAMING is on, yield a partial=True LlmResponse after each FunctionChunk. The FunctionCall uses will_continue=True and partial_args=[PartialArg(string_value=chunk.args)], where string_value is a delta (same as interactions_utils and the Gemini path). No json.loads on the partial path. _finalize_tool_call_response stays the last event. Flag off keeps today's single event.

Did not move LiteLlm onto StreamingResponseAggregator. It is Gemini-shaped (json_path on each PartialArg) and the change would be much larger.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

New tests in tests/unittests/models/test_litellm.py: test_streaming_function_chunks_yield_partials_when_progressive_sse_on and test_streaming_function_chunks_stay_buffered_when_progressive_sse_off. test_generate_content_async_stream_tool_call_includes_aggregated_text now pins the flag on and expects two function-call partials plus the aggregated event.

PYTHONPATH=src .venv/bin/python -m pytest -q tests/unittests/models/test_litellm.py

426 passed.

Reverted lite_llm.py to b0180620 and reran. The flag-on test failed (0 == 3 partials) and test_generate_content_async_stream_tool_call_includes_aggregated_text failed (4 == 6 events). The flag-off test still passed. pytest tests/unittests/{models,a2a,sessions,artifacts} on the fix: 2905 passed, 49 skipped, 3 xfailed.

Manual End-to-End (E2E) Tests:

Not run. No live LiteLLM provider key. The flag-on unit test covers a mock stream of three FunctionChunks plus finish_reason=tool_calls.

Additional context

Claim: #5342 (comment)

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

LiteLlm appended FunctionChunk args until finish_reason. Text already
streamed. Gemini already yields partial function-call events when
PROGRESSIVE_SSE_STREAMING is on. Yield a partial LlmResponse after each
FunctionChunk and keep the aggregated finish event.

Fixes google#5342
partial_args.string_value is a per-event delta in interactions_utils
and Gemini. Joining the buffer on every FunctionChunk duplicated JSON
for consumers that concatenate string_value.

Fixes google#5342
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PROGRESSIVE_SSE_STREAMING is not honored by the LiteLlm adapter — function-call argument deltas are buffered until finish_reason

2 participants