Skip to content

capture: share image history strings across turns to avoid quadratic memory - #1534

Merged
burtenshaw merged 2 commits into
openenv:mainfrom
caoxiaoyuyuyuyuyu:fix/capture-shared-image-history
Nov 5, 2026
Merged

burtenshaw merged 2 commits into
openenv:mainfrom
caoxiaoyuyuyuyuyu:fix/capture-shared-image-history

Conversation

@caoxiaoyuyuyuyuyu

@caoxiaoyuyuyuyuyu caoxiaoyuyuyuyuyu commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

RolloutGraph keeps every turn's request_messages. A chat request re-sends the whole conversation and each request body is JSON-decoded on its own, so turn k holds a fresh copy of every image (and other long text) attached in turns 1..k. For multimodal rollouts graph memory therefore grows quadratically with the number of turns: n turns with one base64 image each retain n(n+1)/2 image copies instead of n.

This PR makes add_turn replace long strings (>= 1024 chars) in request_messages, request_tools and response_message with a single graph-local copy.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • New environment
  • Refactoring

Reproduction (synthetic)

Build a graph of 12 turns where turn k re-sends k+1 user messages, each with a distinct 64 KiB data:image/png;base64,... URL, and every request body is passed through json.loads(json.dumps(...)) as the capture server does. Before this change the graph retains 78 distinct image string objects (~5 MiB); after it, 12 (~0.8 MiB).

Why the output is unchanged

  • Only immutable str objects are shared; equal strings are swapped for an equal string, so every value is identical.
  • Dicts and lists are rebuilt per turn, so message structure is unchanged and mutating one turn's messages cannot affect another turn.
  • The table is per RolloutGraph (not process-global), so it is released together with the graph.

RFC Status

  • Not required (bug fix, docs, minor refactoring)
  • RFC exists: #___
  • RFC needed (will create before merge)

Test Plan

New tests/envs/test_capture_graph_shared_text.py (fully synthetic data):

  • each image is retained once across all turns (object identity), and all turns reference the first copy;
  • editing a nested message field in one turn does not change any other turn;
  • export_session(..., include_messages=True) serialised with json.dumps is byte-identical to the export from the same graph with sharing disabled.

Two of the three tests fail on main and all pass with this change.

PYTHONPATH=src:envs uv run pytest tests/envs/test_capture_graph_shared_text.py -q   # 3 passed
PYTHONPATH=src:envs uv run pytest tests/envs -q -k capture --ignore=tests/envs/test_coding_codeact_env.py   # 191 passed, 26 skipped
uv run usort check <changed files> && uv run ruff format --check <changed files> && uv run ruff check src/ tests/   # clean

(test_coding_codeact_env.py fails to collect locally for an unrelated missing optional dependency.)

Claude Code Review

N/A

🤖 Generated with Claude Code


Note

Low Risk
Memory optimization only; values remain equal strings and container structure is unchanged, with export parity covered by tests.

Overview
Fixes quadratic memory growth in RolloutGraph when multimodal rollouts re-send full chat history each turn. Each decoded request body used to retain its own copy of every prior base64 image and other long string, so n turns could hold n(n+1)/2 copies instead of n.

On add_turn, long strings (≥ 1024 chars) in request_messages, request_tools, and response_message are interned in a per-graph _shared_text table via recursive _share_text; lists and dicts are still rebuilt per turn so edits on one turn do not alias another. Exported session JSON stays the same because only equal immutable strings are shared.

Adds synthetic tests for single retention (object identity), isolation after in-place edits, and byte-identical export_session output with sharing disabled.

Reviewed by Cursor Bugbot for commit a25ef3e. Bugbot is set up for automated code reviews on this repo. Configure here.

Each chat request re-sends the whole conversation and is JSON-decoded on
its own, so turn k holds a fresh copy of every image and long text from
turns 1..k. RolloutGraph kept all of them, which makes memory grow
quadratically with the number of turns in multimodal rollouts.

add_turn now replaces long strings (>= 1024 chars) in request_messages,
request_tools and response_message with one graph-local copy. Only
immutable str objects are shared; dicts and lists are rebuilt per turn,
so values and message structure are unchanged and exports are
byte-identical.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor cursor Bot mentioned this pull request Nov 2, 2026

@burtenshaw burtenshaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified src/openenv/core/harness/capture/graph.py:256 preserves value equality, JSON and Pydantic serialization, exports, training converters, Harbor conversations, live HTML, and nested mutation isolation. Audited every consumer in src/ and envs/; none depends on long-string identity or on mutating ingestion inputs afterwards.

With 12 turns and 64 KiB images, retained image objects fell from 78 to 12, image storage from 5,116,803 to 787,202 bytes, and total traced retained memory from 5,316,205 to 943,662 bytes. The 24- and 48-turn measurements retained 24 and 48 images. Per-turn message containers still grow with history length.

The focused suite passed 316 tests; one optional test skipped. One port-ownership test was excluded after failing identically on main. Changed files pass usort and Ruff formatting/lint. No blocking findings.

Non-blocking suggestion: move tests/envs/test_capture_graph_shared_text.py:18 to tests/core/ in a cleanup. It exercises core code without an environment. Existing capture tests use the current directory and CI collects it.

@burtenshaw
burtenshaw merged commit 505e7a8 into openenv:main Nov 5, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants