Investigate Codex proxy/orchestration OOM from stalled flows and failed prompt-delivery sessions #369

Problem

Recent hard main-process OOMs happened during long multi-agent/Codex-heavy runs. The terminal breadcrumbs immediately before crashes repeatedly showed CodexResponsesAdapter watchdog releases, orchestration prompt-delivery failures, and debug-retention prune bursts.

This issue tracks the most plausible concrete retained-heap root cause candidates. It is a focused child of #327 and #365.

Evidence From Recent Crashes

The retained incident runs include:

• orchestration.prompt_delivery_failed incidents during heavy agent creation.
• Terminal-only lines like:
  • [CodexResponsesAdapter] watchdog releasing proxy-* (attribution=completed silent=...)
  • [debug-retention] pruned ... reason=performance-append ...
• Main-process V8 OOMs with old-space around 2.33-2.73GB.

Primary Suspect: Codex Proxy Flow Retention

packages/codex-headless/src/proxy/CodexResponsesAdapter.ts keeps a flows: Map<string, FlowState>. Flow state can retain:

• decoded buffer
• fullText
• blocks
• tool/reasoning accumulators
• response/request metadata

On every response-chunk, the adapter appends decoded bytes to flow.buffer before deciding whether the flow is active. Secondary flows skip parsing/draining and can retain buffered chunks until response-end or watchdog cleanup.

High-risk behavior found in read-only audit:

• response.completed has cleanup behavior.
• response.failed and response.incomplete finish/publish idle but appear not to clear activeFlowId, mark completed, or delete the flow in the same way.
• That can pin the active slot and demote later real flows, matching the observed pattern of many proxy watchdog releases.

Amplifiers

• responsesProxy.ts mirrors proxy events to JSONL by JSON.stringify and base64-serializing response chunks/request-body evidence. During retry/stall bursts this can create significant transient heap and disk churn.
• orchestration_create_agent can create a child session and then fail bootstrap prompt delivery. On failure it records orchestration.prompt_delivery_failed and returns an agent with promptSubmitted:false, but does not obviously close/kill the newly created session. Each orphan can retain PTY, headless process, rollout watcher, proxy server/adapter, and debug buffers.
• CodexHeadless.freshRolloutCandidates can read and retain rollout candidates while no active rollout is claimed. This is more suspicious when failed-bootstrap/orphan sessions exist.
• Debug retention prune bursts are probably not the primary leak, but can add transient allocations and I/O pressure when proxy/debug dirs are large.

Investigation Targets

In heap snapshots, look specifically for retainers under:

• CodexResponsesAdapter.flows
• FlowState.buffer
• FlowState.blocks
• FlowState.fullText
• ResponsesProxy listeners / mirrored event state
• CodexHeadless.freshRolloutCandidates
• live CodexSession / PTY / headless objects after orchestration.prompt_delivery_failed
• OrchestrationBridge queues or child-session registries

Proposed Fixes To Evaluate

• Release activeFlowId, mark terminal state, and delete flow state on response.failed and response.incomplete, mirroring response.completed cleanup semantics.
• Do not accumulate full buffers for secondary/ignored flows, or cap them by bytes and drop old data.
• Add byte counters for live proxy flow buffers and expose them in incident/perf heartbeat.
• Cap/TTL freshRolloutCandidates and clear on session stop/error.
• On failed create_agent bootstrap prompt delivery, either close the just-created child session or mark it explicitly as failed/orphaned with cleanup policy.
• Add focused regression coverage for failed/incomplete proxy responses releasing the active slot and deleting flow state.

Acceptance

• Reproducing a burst of failed/incomplete Codex Responses flows does not grow CodexResponsesAdapter.flows or retained buffers unboundedly.
• Failed orchestration bootstrap does not leave unmanaged live Codex sessions.
• New counters/events can tell us current flow count, active flow id, secondary-buffer bytes, and failed-bootstrap child lifecycle.
• Heavy multi-agent orchestration no longer shows unbounded main-process heap growth from proxy/orchestration retainers.