Python: fix: prevent superlinear history growth by deduplicating messages in save_messages#7242
Conversation
github-actions
Bot
changed the title
There was a problem hiding this comment.
Pull request overview
This PR fixes a Python history-persistence defect where per-service-call history providers re-persist the full accumulated conversation every round, causing superlinear growth and duplicated context. It introduces message-level deduplication in history providers (in-memory, file, Redis) and adds regression tests to ensure repeated inputs don’t bloat stored history.
Changes:
- Added a message-identity helper and used it to filter out already-persisted messages before appending/pushing.
- Updated
save_messagesbehavior across in-memory, file (JSONL), and Redis history providers to skip duplicates. - Added unit/regression tests covering identical-message deduplication and mixed old/new message lists.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| python/packages/core/agent_framework/_sessions.py | Adds message identity + deduplication to in-memory and file history providers. |
| python/packages/core/tests/core/test_sessions.py | Adds regression tests for deduplication behavior in core providers and looped runs. |
| python/packages/redis/agent_framework_redis/_history_provider.py | Adds message identity + deduplication to Redis history persistence. |
| python/packages/redis/tests/test_providers.py | Adds Redis-provider tests validating deduplication behavior. |
| msg_id = getattr(message, "id", None) | ||
| if msg_id is not None: | ||
| return ("id", msg_id) | ||
|
|
||
| try: | ||
| # Use to_dict() for a stable, deterministic representation | ||
| serialized = json.dumps(message.to_dict(), sort_keys=True, ensure_ascii=False) | ||
| return ("content", message.role, serialized) | ||
| except Exception: | ||
| # Fallback if serialization fails for any reason | ||
| return ("content", message.role, str(message.contents)) |
| existing_identities: set[tuple] = set() | ||
| if file_path.exists(): | ||
| with file_path.open("r", encoding="utf-8") as f: | ||
| for line in f: | ||
| line = line.strip() |
| Uses the message's ID if available, otherwise falls back to a hash of | ||
| its role and serialized contents to prevent duplicate persistence. | ||
| """ | ||
| msg_id = getattr(message, "id", None) |
There was a problem hiding this comment.
this checks for id on Message, which does not exist, message_id does.
| return | ||
| existing = state.get("messages", []) | ||
| state["messages"] = [*existing, *messages] | ||
| existing_identities = {_get_message_identity(m) for m in existing} |
There was a problem hiding this comment.
this has the potential to be a major performance hit, recalculating the id every time and especially since id does not exists, it is doing json.dumps every run, for every message!
| from redis.credentials import CredentialProvider | ||
|
|
||
|
|
||
| def _get_message_identity(message: Message) -> tuple: |
There was a problem hiding this comment.
this should not be replicated in each history provider implementation, rather a single method should be part of the core and that should be used by all, the problem is likely present in each HistoryProvider.
Motivation & Context
The current history providers (
InMemoryHistoryProvider,FileHistoryProvider, andRedisHistoryProvider) blindly append messages on every service call without checking if they already exist in the store.In looped runs (e.g., AG-UI stateless clients or harness todo loops), the transport passes the full accumulated conversation as input on every request. This caused the entire conversation history to be re-persisted every round, leading to superlinear store growth, corrupted context, repeated tool calls, and massive token bloat (up to 3× the real conversation size).
Multi-iteration flows where each service call's message list carries the accumulated conversation rather than a delta, particularly when using stateless chat clients over the AG-UI endpoint.
Description & Review Guide
What are the major changes?
_get_message_identityhelper function that generates a stable identity for a message (usingmessage.idif available, otherwise falling back to a deterministic hash ofrole + serialized contents).save_messagesinInMemoryHistoryProvider,FileHistoryProvider, andRedisHistoryProviderto build a set of existing message identities and filter out duplicates before appending/pushing new messages.What is the impact of these changes?
This hardens the system at the lowest storage level. It completely prevents duplicate message persistence regardless of how the middleware or transport layer constructs the message list. This stops token bloat, prevents context corruption, and ensures graceful degradation in long-running loops without requiring changes to the middleware routing logic.
What do you want reviewers to focus on?
_get_message_identityfallback logic (ensuring it handles missing IDs and serialization edge cases gracefully).RedisHistoryProvider.save_messagesimplementation, specifically ensuring it correctly fetches existing messages vialrangeto build the identity set before pushing only thenew_messagesvia the pipeline.Related Issue
Fixes #7211
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.