Repository navigation
capture: share image history strings across turns to avoid quadratic memory - #1534
Conversation
Each chat request re-sends the whole conversation and is JSON-decoded on its own, so turn k holds a fresh copy of every image and long text from turns 1..k. RolloutGraph kept all of them, which makes memory grow quadratically with the number of turns in multimodal rollouts. add_turn now replaces long strings (>= 1024 chars) in request_messages, request_tools and response_message with one graph-local copy. Only immutable str objects are shared; dicts and lists are rebuilt per turn, so values and message structure are unchanged and exports are byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
burtenshaw
left a comment
There was a problem hiding this comment.
Verified src/openenv/core/harness/capture/graph.py:256 preserves value equality, JSON and Pydantic serialization, exports, training converters, Harbor conversations, live HTML, and nested mutation isolation. Audited every consumer in src/ and envs/; none depends on long-string identity or on mutating ingestion inputs afterwards.
With 12 turns and 64 KiB images, retained image objects fell from 78 to 12, image storage from 5,116,803 to 787,202 bytes, and total traced retained memory from 5,316,205 to 943,662 bytes. The 24- and 48-turn measurements retained 24 and 48 images. Per-turn message containers still grow with history length.
The focused suite passed 316 tests; one optional test skipped. One port-ownership test was excluded after failing identically on main. Changed files pass usort and Ruff formatting/lint. No blocking findings.
Non-blocking suggestion: move tests/envs/test_capture_graph_shared_text.py:18 to tests/core/ in a cleanup. It exercises core code without an environment. Existing capture tests use the current directory and CI collects it.
Summary
RolloutGraphkeeps every turn'srequest_messages. A chat request re-sends the whole conversation and each request body is JSON-decoded on its own, so turn k holds a fresh copy of every image (and other long text) attached in turns 1..k. For multimodal rollouts graph memory therefore grows quadratically with the number of turns: n turns with one base64 image each retain n(n+1)/2 image copies instead of n.This PR makes
add_turnreplace long strings (>= 1024 chars) inrequest_messages,request_toolsandresponse_messagewith a single graph-local copy.Type of Change
Reproduction (synthetic)
Build a graph of 12 turns where turn k re-sends k+1 user messages, each with a distinct 64 KiB
data:image/png;base64,...URL, and every request body is passed throughjson.loads(json.dumps(...))as the capture server does. Before this change the graph retains 78 distinct image string objects (~5 MiB); after it, 12 (~0.8 MiB).Why the output is unchanged
strobjects are shared; equal strings are swapped for an equal string, so every value is identical.RolloutGraph(not process-global), so it is released together with the graph.RFC Status
Test Plan
New
tests/envs/test_capture_graph_shared_text.py(fully synthetic data):export_session(..., include_messages=True)serialised withjson.dumpsis byte-identical to the export from the same graph with sharing disabled.Two of the three tests fail on
mainand all pass with this change.(
test_coding_codeact_env.pyfails to collect locally for an unrelated missing optional dependency.)Claude Code Review
N/A
🤖 Generated with Claude Code
Note
Low Risk
Memory optimization only; values remain equal strings and container structure is unchanged, with export parity covered by tests.
Overview
Fixes quadratic memory growth in
RolloutGraphwhen multimodal rollouts re-send full chat history each turn. Each decoded request body used to retain its own copy of every prior base64 image and other long string, so n turns could hold n(n+1)/2 copies instead of n.On
add_turn, long strings (≥ 1024 chars) inrequest_messages,request_tools, andresponse_messageare interned in a per-graph_shared_texttable via recursive_share_text; lists and dicts are still rebuilt per turn so edits on one turn do not alias another. Exported session JSON stays the same because only equal immutable strings are shared.Adds synthetic tests for single retention (object identity), isolation after in-place edits, and byte-identical
export_sessionoutput with sharing disabled.Reviewed by Cursor Bugbot for commit a25ef3e. Bugbot is set up for automated code reviews on this repo. Configure here.