Subagent harness: async background agents, resumable contexts, deterministic orchestration, worktree isolation, and fleet observability

[justrach]

Subagent harness: async background agents, resumable contexts, deterministic orchestration, worktree isolation, and fleet observability

Motivation

The compare-strategies core — spawn N variants, judge them, promote a winner — is what makes this project different from a plain agent loop, and the workflow tool's phases/pipeline split already gets the fan-out semantics right. But the subagent runtime underneath it is still built for "one blocking parallel batch per turn," and that ceiling is now the thing limiting how far the tournament model can scale: bigger fleets, cheaper judging, safer parallel edits, and orchestration that a script — not just the root LLM's own tool calls — can drive.

The worktree gap is concrete, not hypothetical: in a real 5-parallel-builder session sharing one checkout, sibling agents raced on bun.lock, and one agent ended up hand-rolling git worktree add mid-task just to branch/commit/push safely while its siblings kept editing the same tree. -w already solves this at the session level; it just doesn't reach down to the individual agents inside one session's fan-out.

Where CodeGraff stands today

#	Capability	Status	Evidence
1	Async background subagents (fire-and-forget, orchestrator keeps working, structured completion + usage events)	PARTIAL	runTools spawns every external call — including subagents — via io.async and then blocks on fut.await for all of them before the turn can continue (src/agent_tools.zig:89-155, esp. 152-153). execWorkflow/runPipeline do the same per phase/chain (src/workflow.zig:184-187, 302-307). The fire-and-forget primitive already exists, just not for agents: background bash jobs (bash {run_in_background:true}, poll via bash_output, bash_kill) run on io.concurrent and are pumped/reaped independently of the turn (src/jobs.zig:304-563, src/schema.zig bash_output/bash_kill specs). N-agents-in-one-call already works (subagent_spec description: "call several times in one response"; workflow phases/pipeline arrays) — it's the non-blocking half that's missing, plus no usage stats (tokens/tool-calls/duration) ever ride along as a discrete event.
2	Resumable agent contexts (same identity, follow-up messages, full context kept)	ABSENT (for subagents)	runSub builds a brand-new Agent with a fresh arena and empty history every call, seeds it with just the one prompt, runs it to completion, and frees the arena the moment it returns (src/subagent.zig:157-219, defer arena_state.deinit() at 164). There is no id/handle that lets a later tool call address "the same subagent" again. Root-session resume does exist (/save//resume, src/session.zig, src/session_run.zig; the TS SDK's Harness.session()) but that's the one root conversation, not a spawned child. docs/hyperagents.md §8 lists "Depth > 2" as explicitly not built, and depth 1 is exactly where multi-round delegation to a child would need to live.
3	Deterministic orchestration scripts (agent()/parallel()/pipeline(), concurrency caps + queueing, loop-until-dry, adversarial-verify, shared spent()/remaining() budget, on-disk run journal + cached resume)	PARTIAL	The shape is already right: workflow's phases mode is fan-out-with-barrier and pipeline mode is per-item-with-no-barrier — the exact parallel()/pipeline() split (src/workflow.zig:1-8, 62-68, execWorkflow 207-364, runPipeline 144-201). Concurrency caps + queueing already exist: RunBudget.acquireConcurrency admits up to max_concurrency (default 8) and spin-waits for a free slot rather than erroring (src/run_budget.zig:52-67). A budget primitive with exactly used()/remaining() already exists (src/run_budget.zig:97-104) — but it counts model calls, not tokens, and it's internal Zig state, not reachable from a script. What's missing: this only runs as one JSON blob the root LLM decides to emit as a tool call, not as agent()/parallel()/pipeline() functions a TS/Python script calls directly; sdk/ts/harness.ts and sdk/py/harness_sdk.py expose only ask/chat/session today, nothing for orchestration. Loop-until-dry and adversarial-verify: no such primitive (closest is execWorkflow's single automatic retry-once on failure, src/workflow.zig:311-336). Run journal + cached resume: absent — the trajectory JSONL (.graff/trajectories/<run-id>.jsonl) is an append-only lineage log, not a keyed cache a re-run can consult. docs/hyperagents.md §3 says this outright: "the loop itself belongs in SDK code (Python/TS), not in the harness."
4	Schema-validated structured returns, validated + retried at the harness layer, no prose parsing	ABSENT	The one signal-extraction path in the whole scoring/judging pipeline is prose scraping: parseEvalScore scans free text for a score: <N> / score=<N> line (src/repl_glue.zig:112), used by the eval harness (src/agent_eval.zig:40, 212) and by workflow's own variant judge (src/subagent.zig:362). Tool input schemas exist and are solid (src/schema.zig ToolSpec/JSON Schema per tool) — but that validates what the orchestrator sends a tool, not what a subagent's final report contains. No per-agent output schema, no harness-side validation, no auto-retry-on-violation loop.
5	Per-agent isolation: opt-in git-worktree-per-agent, auto-removed if unchanged	PARTIAL	Worktree isolation is real, just scoped to the whole session, not per-parallel-agent: --worktree/-w <name> creates .graff/worktrees/<name> on branch worktree-<name>, chdirs the entire process into it, and auto-commits each turn (src/session_start.zig:74-124 setupWorktreeAndBanner; src/jobs.zig:140-302 worktreeAutoCommit/worktreeCommand; graff worktree list/merge/remove/prune in src/cli.zig:66-92). Because the isolation is a process-wide chdir, every subagent fanned out from one workflow/subagent call — via io.async — still shares that one cwd; runSub builds each child Agent straight from the parent's ctx.* with no per-child working directory at all (src/subagent.zig:157-188). This is exactly the shared-checkout race described above.
6	Heterogeneous fleets: per-agent model + reasoning-effort override, typed registry with scoped toolsets	PARTIAL	The typed registry exists and is good: builtin_agent_types (reviewer/researcher/implementer/skeptic) plus a two-tier .harness/agents/*.md override (personal ~/.harness/agents, private ./.harness/agents), selected with agent: "<name>" on any subagent/workflow task (src/fleet.zig:33-95, 251-257, 357-372; subagent_spec/workflow_spec in src/schema.zig:202-216). But it's a persona registry only — every type gets the identical toolset (bash, read_file, edit_file, write_file, webfetch, codedb; src/schema.zig base_specs), so "researcher — never edits" is prompt text, not an enforced boundary. Per-agent model/effort override is absent entirely: runSub hands the child .provider = ctx.provider verbatim (src/subagent.zig:172), and neither subagent_spec nor a workflow task's schema has a model or effort field (confirmed against src/schema.zig:202-216) — model and reasoning effort are process-wide, changed only via the root's /model//effort. An expensive orchestrator + cheap parallel builders can't be expressed in one session today.
7	Observability: complete per-agent JSONL trajectory to disk, orchestrator context gets only the final report, per-agent usage accounting	PARTIAL	Context protection is real and already matches the target exactly: a subagent's full turn/tool history never reaches the orchestrator — only its final text plus an inspect: <path> pointer comes back as the tool result (src/subagent.zig:249-261). Two logging layers already capture a lot: the shared per-run Tracer (.graff/traces/<run-id>.jsonl) records every API and tool call from every agent with an agent label, latency, and token/cache counts (src/trace.zig:101-166); the per-run Trajectory/DGM archive (.graff/trajectories/<run-id>.jsonl) writes one summary node per subagent run — kind, label, prompt fingerprint, tools used, ok, ms, context tokens (src/subagent.zig:225-242, src/trace.zig:180-249); cards.writeSubagentDetail additionally writes a per-subagent markdown report under .graff/subagents/ (src/cards.zig:188-218). Gap: these are per-run files shared by every agent in the invocation, not one dedicated trajectory file per individual subagent, and no token/cost usage rides along on the returned tool result or a structured event — only ms and the tool-name list make it back to the caller.
8	Cross-cutting guardrails: pre-tool-call hooks bound to all subagents automatically, unskippable	EXISTS	pre_tool/post_tool/turn_end hooks (.harness/settings.json, src/hooks.zig:38-96) run through one choke point every external tool call passes through — execTool (src/exec.zig:69-71) — which calls hookGate (src/tools.zig:213-229: exit 2 blocks, stderr surfaces to the model) before dispatch. Subagent tool calls are fanned out through that same execTool (src/agent_tools.zig:152), so a pre_tool guard applies to every child agent structurally, with no per-agent opt-out. This one is already in good shape.

Proposals

P0 — highest leverage for the tournament/compare-approaches core

1. Per-agent git-worktree isolation, auto-removed if unchanged

Extend the existing -w/graff worktree mechanism down from "whole session" to "one worktree per fanned-out agent," opt-in per subagent/workflow-task call. Directly removes the shared-checkout race.

• Build: a worktree: true (or isolate: "worktree") field on the subagent tool and on workflow phase/pipeline task objects; runSub resolves a scratch dir (.graff/worktrees/agent-<sub_id> off worktreeCommand's existing add/remove path in src/jobs.zig) and threads it through as the child Agent's cwd instead of inheriting ctx.* verbatim (src/subagent.zig:157-188). On return, diff the worktree against its base branch: no changes → remove it immediately (git worktree remove); changes → leave it for the orchestrator to land (mirrors graff worktree merge) or auto-land if the caller asked for it.
• API sketch (Zig core):

  // subagent.zig
  pub fn runSub(ctx: ToolCtx, kind: []const u8, label: []const u8, prompt: []const u8,
      sys_override: ?[]const u8, niche: []const u8, isolation: Isolation) !ToolOutput
  
  const Isolation = enum { shared_cwd, worktree };

  // sdk/ts/harness.ts — new field on the existing subagent/workflow task shape
  interface SubagentTask { description: string; prompt: string; agent?: AgentTypeName;
    isolation?: "shared_cwd" | "worktree" }

• Acceptance criteria: N parallel subagent/workflow-phase tasks with isolation:"worktree" never touch the same working tree; an unchanged worktree is removed with no trace left in git worktree list; a changed one survives with a clean graff worktree merge <id> path; -w at the session level and isolation:"worktree" per-agent compose (nested worktrees) without collision.

2. A deterministic orchestration layer in the SDKs: agent()/parallel()/pipeline() + a shared budget + a run journal

The harness already has the right shape internally (workflow's phase/pipeline split, RunBudget.used()/remaining()); promote it to first-class SDK primitives instead of leaving it reachable only through one root-LLM tool call, per docs/hyperagents.md §3's own framing that "the loop itself belongs in SDK code."

• Build: extend sdk/ts/harness.ts / sdk/py/harness_sdk.py with functions that drive the existing --json protocol and workflow/subagent tools programmatically rather than through the root model — parallel() maps to a workflow phase (barrier), pipeline() maps to workflow's pipeline mode (no barrier), agent() is a single subagent call. Back it with an on-disk run journal (.graff/journal/<run-id>.json, keyed by a hash of {prompt, sys_override, agent-type, isolation}) so a re-invoked script with an unchanged call reuses the cached result instead of re-running it — this is what makes re-judging a tournament cheap. Surface RunBudget.used()/remaining() (already implemented, just internal) over the --json/serve protocol as a budget event so a script can branch on spent()/remaining() mid-run, and extend RunBudget to track tokens in addition to call count.
• API sketch:

  // sdk/ts/harness.ts
  export async function agent(spec: SubagentTask, opts?: RunOptions): Promise<AgentResult>;
  export async function parallel(tasks: SubagentTask[], opts?: RunOptions): Promise<AgentResult[]>;
  export async function pipeline(items: string[], stages: StageSpec[], opts?: RunOptions): Promise<AgentResult[]>;
  export interface RunOptions { journal?: boolean; budget?: { maxTokens?: number; maxCalls?: number } }
  export interface Budget { spent(): { calls: number; tokens: number }; remaining(): { calls: number; tokens: number } }

  # sdk/py/harness_sdk.py
  def agent(spec: SubagentTask, *, journal: bool = True) -> AgentResult: ...
  def parallel(tasks: list[SubagentTask], *, budget: Budget | None = None) -> list[AgentResult]: ...
  def pipeline(items: list[str], stages: list[StageSpec]) -> list[AgentResult]: ...

• Acceptance criteria: a tournament script (spawn K strategies via parallel(), judge, re-run only the tasks whose inputs changed) produces byte-identical results to today's workflow tool call on a fresh journal, and skips re-running unchanged calls on a warm one; a budget={maxTokens: N} run stops admitting new parallel()/pipeline() work once remaining().tokens hits 0 rather than erroring mid-fan-out.

3. Async, fire-and-forget subagents with structured completion events

Extend the pattern that already exists for background bash jobs (src/jobs.zig spawnJob/jobPump/jobOutput/jobKill, run_in_background on bash) to subagent/workflow tasks, so the orchestrator can launch a fleet of tournament strategies and keep working instead of blocking on runTools' fut.await loop (src/agent_tools.zig:152-153).

• Build: a run_in_background: true option on subagent, mirroring bash's; it returns an agent id immediately (via the same Job-style registry, src/jobs.zig:309-336) instead of blocking; new meta tools agent_output/agent_wait (poll or block for completion), returning { result, usage: { input_tokens, output_tokens, cache_read_tokens, tool_calls, duration_ms } } — sourced from the per-agent numbers runSub already computes (run_ms, used_tools, agent.effectiveContextTokens(), src/subagent.zig:220-242) but currently only writes to the trace file instead of returning them.
• API sketch:

  // subagent.zig
  pub fn spawnSubBackground(ctx: ToolCtx, /* same args as runSub */) !u64; // returns agent id
  pub fn subAgentStatus(id: u64) SubAgentStatus; // running | done(AgentResult) | failed

  // --json protocol: new event type
  { "type": "agent_finished", "id": "sa-014-abcd", "ok": true,
    "usage": { "input_tokens": 1820, "output_tokens": 340, "tool_calls": 6, "duration_ms": 4110 } }

• Acceptance criteria: launching 5 background subagents returns control to the orchestrator immediately; agent_output/agent_wait (or the --json agent_finished event) deliver the result plus usage stats without the orchestrator ever seeing the child's intermediate turns; a synchronous subagent call (no run_in_background) behaves exactly as today, unchanged.

P1

4. Schema-validated structured returns, validated and retried at the harness layer

Replace the parseEvalScore prose-scraping path (src/repl_glue.zig:112) — the sole score-extraction mechanism feeding the fleet/DGM scoring loop — with an opt-in JSON Schema on a subagent's final report, validated by the harness (not the caller) with automatic retry on violation.

• Build: an optional result_schema field (raw JSON Schema string, same convention as ToolSpec.schema in src/schema.zig) on subagent/workflow-task input; on completion, runSub asks the child for one more turn constrained to that schema (reusing the existing tool_choice-forcing / strict-mode machinery already built for attempt_completion, src/schema.zig:159-192, architecture.md "Every message is a tool"), validates the JSON against the schema, and retries up to N times before surfacing a schema-violation error instead of silently returning prose.
• API sketch:

  pub fn runSub(ctx: ToolCtx, kind: []const u8, label: []const u8, prompt: []const u8,
      sys_override: ?[]const u8, niche: []const u8, result_schema: ?[]const u8) !ToolOutput

  interface SubagentTask { /* ... */ resultSchema?: object } // AgentResult.data typed via the schema

• Acceptance criteria: a workflow variant-judge task with result_schema: {score:number, reasons:string[]} never returns free text to the caller — either a schema-valid JSON object or an explicit is_error after N failed retries; variantJudgePrompt/scoreVariants (src/subagent.zig:288-403) switch from parseEvalScore to reading .score off the validated object.

5. Resumable agent contexts (multi-round delegation to the same identity)

Let a completed subagent be re-addressed with a follow-up message, keeping its arena and message history alive instead of freeing them on return (src/subagent.zig:163-219).

• Build: an id returned alongside a subagent's result (the existing sub_id from cards.subagentId, src/subagent.zig:209-211, already threads through); a new resume field on subagent (resume: "<sub_id>", prompt: "<follow-up>") that looks the id up in a session-scoped live-agent map instead of constructing a fresh Agent, appends the follow-up as a new user turn, and re-runs. Requires deferring the arena free until the agent is explicitly released (idle-timeout or an explicit close param) rather than on every runSub return.
• API sketch:

  pub var g_live_subagents: std.AutoHashMap(u64, *Agent) = .empty; // keyed by sub_id ordinal
  pub fn resumeSub(ctx: ToolCtx, id: u64, prompt: []const u8) !ToolOutput

  await agent(spec);                 // -> { id, result }
  await agent({ resume: id, prompt: "now re-check the edge cases" }); // same identity, full context

• Acceptance criteria: two calls with resume: <id> see the same message history a single long-lived agent would (verified via agent.messages.items.len growth, not a fresh empty array); re-judging a tournament winner across two rounds costs one context instead of two.

6. Heterogeneous fleets: per-agent model/effort override + scoped toolsets on the typed registry

• Build: extend subagent_spec/workflow task schemas (src/schema.zig:202-216) and fleet.AgentType (src/fleet.zig:33-39) with optional model/effort fields, resolved to a per-child Provider in runSub instead of the hardcoded .provider = ctx.provider (src/subagent.zig:172); extend .harness/agents/<name>.md frontmatter with a tools: allowlist enforced structurally in execToolInner's subagent gate (architecture.md "Subagents run on pool threads... gated structurally in execToolInner"), not just described in the persona prompt.
• API sketch:

  ---
  name: implementer
  model: claude-opus-4-8
  effort: high
  tools: [bash, read_file, edit_file, write_file, codedb]
  ---
  

  await agent({ agent: "researcher", prompt: "...", model: "deepseek-v4-mini" }); // inline override

• Acceptance criteria: a tournament with an expensive judge-tier orchestrator and cheap builder-tier parallel workers runs both tiers in one session; a researcher-typed agent's write_file call is denied at the gate (not just discouraged by prompt), independent of /yolo.

P2

7. Per-agent dedicated trajectory files + usage-bearing completion cards

Split the per-run Tracer/Trajectory files (src/trace.zig:101-249) into one JSONL per subagent (.graff/trajectories/<run-id>/<sub_id>.jsonl) in addition to (not instead of) the run-level rollup, and attach agent.effectiveContextTokens()/cache-read counts to the object cards.writeSubagentDetail already builds (src/cards.zig:188-218) so token/cost numbers travel with the result the orchestrator sees, not just with ms and tool names.

• Acceptance criteria: cat .graff/trajectories/<run-id>/<sub_id>.jsonl reconstructs one agent's full turn-by-turn trace standalone, with no run_id-wide grep needed; the completion card printed for a subagent includes input/output/cache token counts alongside duration.

8. (No new proposal — already solid.) The pre_tool/post_tool hook choke point (src/exec.zig:69-71, src/tools.zig:213-243) already binds to every subagent unconditionally; P0 items 1-3 above should route their new code paths (background-agent spawn, worktree assignment) through the same execTool/hookGate boundary rather than adding a side door, so this guarantee doesn't quietly regress as the runtime grows.

[justrach]

Field notes from implementing this issue: delegation trust is itself a harness gap

While landing P0 #1 (worktree isolation, 8e59e9b on feat/276-subagent-harness) and attempting P0 #3, we hit a subagent failure mode this issue doesn't yet cover — worth capturing because it argues for a new proposal.

Setup: the orchestrating session is rooted in an unrelated repo, delegating work into this one via absolute paths and a dedicated worktree (exactly the multi-repo delegation CodeGraff fleets will do). Same task, same worker model tier, five attempts:

Attempt	Prompt framing	Outcome
P0 #1 (baseline)	Plain task + facts, no trust narrative	Worker independently verified premises (repo, issue, branch), then shipped. 236/236 tests.
P0 #3, try 1	Same plain framing	Refused as suspected injection: cross-repo mismatch + tool-policy hooks read as "hallmarks"; notably made zero verification calls before refusing.
try 2 (resume + evidence rebuttal)	Point-by-point checkable evidence	Accepted after verifying — then lost to an infra stall (stream watchdog), no partial state thanks to the clean worktree.
try 3 (resume)	"Continue"	Second stall, again mid-read, again zero corruption.
try 4 (fresh)	Task + defensive legitimacy preamble	Refused — cited the preamble itself ("proceed without further deliberation") as the injection tell. Zero verification calls.
try 5 (fresh)	Back to plain factual framing	In flight at time of writing.

Two observations that generalize:

1. Workers judge legitimacy from narrative style, not verifiable state. The refusing workers had every means to check the facts in seconds (git log on the target branch, the issue on this repo, the authenticated account being the repo owner) and did not run a single command; the accepting workers ran the checks unprompted. And over-arguing legitimacy in prose made refusal more likely, not less. Conclusion: delegation legitimacy should not travel as prose at all.

2. Cheap resumability matters as much as spawning. Both stalls cost zero work only because isolation kept the worktree clean and the task was re-issuable. With P0 Harden ask_user JSON protocol and GUI transcript #3's completion events plus a run journal (P0 Add Zig-native GUI live JSON controls #2), a stalled worker becomes a resume, not a redo.

Proposal to add (suggest P1) — structured delegation provenance: spawn metadata should carry a machine-checkable provenance block — session owner identity, target repo + branch, authorizing artifact (issue/commit refs), and the explicit scope granted — which the harness verifies (or exposes one verify call for) before the worker's first token. Workers then apply their skepticism to a checklist with pass/fail results instead of judging the vibes of a prompt. This composes with the fleet persona registry: personas could declare how much provenance they require before accepting cross-repo scope.

[justrach]

Shipped in v0.0.216 — f657907 (per-agent worktree isolation), 96a2ca9 (async background subagents + agent_output), b04ae02 (TypeScript orchestration layer with token budgets and journaled resume), merged via #279. Follow-ups from the same area are tracked separately in #295 (worktree scoping across dependent stages) and #296 (pipeline stage scoring).