Skip to content

feat(tools): tool results return handles by default above a size threshold (#440) - #451

Merged
justrach merged 1 commit into
release/0.0.242from
feat/440-handles-by-default
Aug 6, 2026
Merged

feat(tools): tool results return handles by default above a size threshold (#440)#451
justrach merged 1 commit into
release/0.0.242from
feat/440-handles-by-default

Conversation

@justrach

@justrach justrach commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Closes #440. The generalization of #409, and the core of graff's answer to "context as variables" without adopting a Python kernel: the filesystem is the namespace, bash is the REPL, both of which graff already has.

The finding this fixes

A live A/B eval showed #409's spill path is effectively dead code for the local toolset. Three caps sat in series, and layer 2 reduced every execTool result to 2,000 chars, so layer 3's ~136 KB send-time threshold was unreachable for bash/read_file/webfetch/codedb. Across 12 runs against a 168 KB log, including a forced cat, the spill never fired once.

The contract

Above a threshold, a tool result returns, uniformly and at tool time: a bounded preview, a durable handle (absolute path), the byte count, and a one-line shape hint (line count for text, top-level keys for JSON). The model slices what it needs with read_file/grep/bash instead of carrying the payload turn after turn.

What happened to the three caps

layer verdict
tool_preview_chars 2000 + persistToolResult Removed, replaced by tool_handle.zig. It was the handle idea without the payload information that makes a handle actionable: path only, no size, no shape. Same on-disk namespace, so nothing is orphaned. Paths are now absolute, which also fixes a latent bug where a worktree-isolated subagent (#276) could never open the old relative path.
bash_stdout_cap 128 KiB Demoted to a capture ceiling and raised to 1 MiB; codedb's 64 KiB context truncation removed. Both destroyed bytes purely to protect context, which the handle now does non-destructively. At 128 KiB the handle would have lied for exactly the measured case: a quarter of that 168 KB log was already gone before anything could be written down. runCapped buffers incrementally, so this is a ceiling, not an allocation.
perOutputCap / capOversizedToolOutputs / tool_spill Kept, narrowed to a documented backstop. effectiveThreshold(perOutputCap()) clamps the tool-time threshold at or below the send-time cap, so the two are ordered by construction rather than stacked. What can still reach the backstop is a session resumed from a pre-#440 build, whose stored outputs never met this contract; deleting the pass would mean destroying those bytes.

Threshold default is 4096 bytes, knob GRAFF_TOOL_HANDLE_BYTES (0/unparseable ignored, so the contract cannot be silently disabled). Verified to sit above the tier-2 eval case whose 4001-byte outputs must stay whole to drive compaction. Prompt guidance ships as a capability-gated segment per #421, so it contributes zero bytes under --no-local-tools.

Verification

zig build test1048/1048, exit 0 (baseline 1043; +6 new, −1 removed, which proves the new module's tests are actually reachable). scripts/eval-tier1.sh fully green.

Beyond unit tests, this was driven end to end through the real harness against a mock Codex: a 20 KB read_file came back as a handle with preview ≤ 4096 bytes, an absolute path, the exact byte count, and shape hint 1 lines, and the on-disk file was byte-for-byte equal to the payload.

Behaviour changes worth knowing

  1. The marker text changed. Anything parsing [full tool result: … — inspect with read_file] breaks; the only in-repo consumer was updated.
  2. Handle paths are absolute rather than .graff/tool-results/... relative.
  3. Per-result preview budget goes 2000 → 4096 bytes, and is now constant per result.
  4. A bash command emitting 128 KiB–1 MiB no longer reports [stdout truncated at 128 KB]; it yields a complete handle.
  5. .graff/tool-results/ can grow faster. Bounded by a new 64 MiB per-process budget with an honest truncation marker, but there is still no GC sweeper for that directory. Pre-existing gap, filed separately.

Not measured: the 1 MiB capture ceiling's memory behaviour under real load is reasoned from incremental reads, not benchmarked. src/session_run.zig sits at 599/600 lines, so the next change there needs an extraction first.

## What changed

`src/tool_handle.zig` is the one size contract for a tool result. At or below
the threshold (4 KiB, `GRAFF_TOOL_HANDLE_BYTES`) a result reaches the model
exactly as the tool produced it. Above it, the complete bytes are written under
`.graff/tool-results/` and the model gets a handle instead: a bounded preview,
that file's ABSOLUTE path, the byte count, and a one-line shape hint measured
from the payload (line count, or the top-level keys of JSON, only after the
document scans clean end to end). Meta tools never touch it — runTools applies
it to the external half of a batch.

This collapses three caps that used to sit in series:

- `agent_tools.persistToolResult` + `toolPreviewText` (2000 chars + a bare
  path) are REMOVED. They were the handle idea without the information that
  makes a handle usable: the model was told where the bytes were and nothing
  about what was in them, so it could not decide what to ask for without
  re-running the tool.
- `tools.bash_stdout_cap` is DEMOTED from a context cap to a capture ceiling
  and raised 128 KiB -> 1 MiB, and `codedb`'s 64 KiB result truncation is
  REMOVED outright (the const survives as the compact-read capture bound).
  Both existed to protect context by destroying bytes; the handle protects
  context harder, non-destructively. At 128 KiB the handle would have lied for
  exactly the case that motivated the issue — a 168 KB log, a quarter of it
  already gone before anything could be written down.
- `capOversizedToolOutputs` / `perOutputCap` / the #409 spill are KEPT, and
  narrowed to a backstop. `tool_handle.effectiveThreshold` clamps the tool-time
  threshold at or below `perOutputCap()`, so a result this process produced can
  no longer reach the send-time pass oversized. What still can is history this
  process did not produce — a session resumed from a pre-#440 build — which is
  precisely where deleting the pass would mean destroying bytes instead.

A capability-gated prompt segment (`prompt_text.tool_handle_note`, gate
`.local_tools`) tells the model the contract exists. It ships only when the
tools that produce a handle and the tools that can open one are both present:
under `--no-local-tools` a path is not something the session can act on.

## Why

- Problem/failure mode: the send-time spill (#409) was unreachable for the
  local toolset. Every `execTool` result was already reduced to 2000 chars at
  tool time, so nothing ever arrived at the ~136 KB send-time threshold — in 12
  live runs against a 168 KB log, including a forced `cat`, it never fired
  once. Meanwhile the bytes that mattered had already been destroyed twice
  over, by the 128 KiB bash cap and the 64 KiB codedb cap.
- Reason for this approach: a cap only rescues bytes at the moment of overflow.
  Handles-by-default means a large result never occupies context in the first
  place, and one result costs one threshold however large it is. The filesystem
  is the namespace and bash is the REPL, both of which graff already has, so
  this needs no kernel and no new tool.
- Constraints or trade-offs: the preview budget per oversized result roughly
  doubles (2000 -> 4096 bytes), which is the price of a head worth reading next
  to a marker worth acting on; it is constant, where the 128 KiB / 64 KiB /
  136 KiB numbers it replaces were not. A handle is written whole or not at
  all, bounded by a per-process budget, because a partial one would make the
  byte count a lie.
- Rejected alternatives: deleting the send-time cap as dead code. It is dead
  for anything runTools produced, and only for that; a resumed old session
  still carries outputs that never met this contract, and for those the
  alternative to the pass is an overflow.

Co-Authored-By: Codegraff <blackfloofie@codegraff.com>
@justrach
justrach changed the base branch from main to release/0.0.242 August 6, 2026 12:31
@justrach
justrach merged commit 4a0148d into release/0.0.242 Aug 6, 2026
6 checks passed
@justrach
justrach deleted the feat/440-handles-by-default branch August 6, 2026 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tool results return handles by default above a size threshold (filesystem as the namespace)

1 participant