Skip to content

fix(t3team): a dead agent turn can no longer settle the askAgent step as its answer - #284

Merged
johnnyelwailer merged 1 commit into
mainfrom
feature/host-failed-agent-turn-swallowed-as-step-completion-8429c93c
Sep 16, 2026
Merged

johnnyelwailer merged 1 commit into
mainfrom
feature/host-failed-agent-turn-swallowed-as-step-completion-8429c93c

Conversation

@johnnyelwailer

Copy link
Copy Markdown
Owner

A failed agent turn silently completed the askAgent step

Problem

When a workflow's thread.askAgent (agent step) turn died without completing — the provider session exited mid-stream (session.exited), or the turn was aborted before it finished (host turn-watchdog, provider abort) — the step settled with whatever the agent had streamed up to that point. The truncated preamble was reported to the run as the step's full answer, and the run continued. The documented failure path (bounded re-drive → "needs attention" with the provider's reason) only engaged when the session ended in status "error".

Reproduced live: a gateway stream break left the step's thread mid-turn; the step then "completed" on the one preamble message the agent had emitted before the break.

Root cause (three seams, all host-side)

  1. The turn tracker only treated "error" as a dead turn. In t3team-workflowTurnResolution.ts, noteSession marked the watch as failed only when the session write that ended the wait had status "error". The "stopped" (session exited) and "interrupted" (turn aborted) writes ended the wait but settled the ask as "answer" with the last streamed candidate — the partial text.
  2. The re-drive's "existing answer" lookup had the same blind spot. findCompletedAnswer (used by the interrupted-turn re-drive and its invariant-retry path) returned the first completed assistant message after the prompt — which, for a dead turn, is its preamble. Even a correctly rejected turn could have been re-consumed as an answer.
  3. Merge artifact in the ingestion layer. ProviderRuntimeIngestion.ts carried two case "turn.aborted" entries: upstream's → "interrupted" (fix(server): unblock OpenCode approvals and stop pingdotgg/t3code#9653) and a fork-only → "ready" (shadowed dead code). The fork test pinning "ready" had been failing on main since the merge; the shadow made the live behavior unreadable.

Fix

  1. Host contract: any dead-state write ends the ask as a failure. A session write that ends the wait with status error / stopped / interrupted settles the ask as failed with a status-specific reason (or the write's lastError when it carries one). The reactor's existing bounded re-drive then fires; when the budget is spent the run fails with the provider's reason. Only a ready/idle write — the turn actually completed — may settle with text. A dead turn's streamed text is preamble, never an answer.
  2. The lookup refuses a dead turn's preamble. findCompletedAnswer skips messages belonging to the thread's latest turn when that turn ended interrupted/error (the projection settles dead turns exactly that way). A completed answer from an earlier turn still counts.
  3. Ingestion cleanup. Removed the shadowed case "turn.aborted": return "ready"; the live "interrupted" case now documents why it is the right status. The stale fork test is updated to pin "interrupted" (matching upstream #9653's OpenCode tests, which already asserted it).

Why this is safe

  • User stops are untouched. Stopping a step's turn calls tracker.forget and stops the run — the step never settles at all. The dead-state rule only engages when the host is still waiting on the turn.
  • Re-drive already handled dead sessions. The decider's thread.turn.resume explicitly accepts a thread whose last turn ended interrupted/error ("a partial assistant message may trail the user message, but the reply never completed"). The session-level transient-retry ladder and the host's step-level re-drive coordinate through the decider's one-active-turn invariant.
  • interrupted is not "dead" in the wait rule. An interrupted session is still alive: a stray interrupted write for a turn the watch never saw running does not end the wait (dead statuses error/stopped still end it unconditionally).

Tests

  • t3team-workflowTurnResolution.test.ts: stopped now settles failed (with/without partial text); new interrupted cases (pre-abort text refused, lastError honored, stray-write guard).
  • t3team-workflowTurnAnswerLookup.test.ts: dead-latest-turn preamble never counts; a completed reply still counts when only a later turn died.
  • t3team-workflowEngineTurnAnswer.integration.test.ts (host-level regression, both dead shapes): a turn that dies mid-stream — or is aborted — rejects the askAgent step and re-drives it; the run completes only with the re-driven turn's answer, and the rejection is journaled as a re-drive attempt on the run row. Pre-existing suites (preamble-not-answer, no-text failure, restart re-drive, budget exhaustion) all still pass.
  • ProviderRuntimeIngestion.test.ts: the abort test now pins "interrupted" (was failing on main).

Verification

  • Focused vitest: t3team-workflowTurnResolution.test.ts, t3team-workflowTurnAnswerLookup.test.ts, t3team-workflowEngineReactorTasks.test.ts, t3team-workflowEngineTurnAnswer.integration.test.ts, ProviderRuntimeIngestion.test.ts, plus decider/transient-retry/rehydrate/registry/reactor suites — all green.
  • tsgo --noEmit in apps/server: no errors in the changed files (the remaining diagnostics are pre-existing on main, in untouched files).
  • vp lint on all changed files: clean. node t3team-additive-guard.mjs: passes.

Related

  • GHE pj/nexi-distribution Fix startup V8 OOM: bound the decider's command read model at boot #297 — turn-layer half: a mid-stream provider death emits no terminal event, so the turn only settles when the 10-minute watchdog interrupts it. This PR makes the host robust to that death however it arrives; the turn layer should still emit a terminal event promptly.

… as its answer

A turn that died without completing (provider session exited mid-stream,
or the turn was aborted by the host watchdog / provider) settled the
step's ask with whatever the agent had streamed before the death — the
truncated preamble was reported as the step's full answer, and the run
continued. Only a session ending in "error" engaged the bounded
re-drive → needs-attention path.

- Turn tracker: any dead-state write that ends the wait (error /
  stopped / interrupted) now settles the ask as "failed" with a
  status-specific reason (or the write's lastError), so the reactor's
  bounded re-drive fires and, when the budget is spent, the run fails
  with the provider's reason. Only a ready/idle write (the turn
  actually completed) may settle with text.
- findCompletedAnswer: messages belonging to the thread's latest turn
  no longer count as a completed answer when that turn ended
  interrupted/error — the re-drive cannot consume a dead turn's
  preamble.
- ProviderRuntimeIngestion: removed the shadowed second
  `case "turn.aborted"` (→ "ready", unreachable dead code left by the
  2026-09 merge); the live "interrupted" case documents why the
  status must not be "ready". The stale fork test pinning "ready"
  (failing on main since the merge) now pins "interrupted", matching
  upstream #9653's OpenCode tests.

Regression tests: host-level integration tests pin that a mid-stream
dead turn (stopped) and an aborted turn (interrupted) reject the
askAgent step, re-drive it, and complete the run only with the
re-driven turn's answer; tracker and lookup unit tests pin the
per-status settlement.
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:M labels Sep 16, 2026
@johnnyelwailer
johnnyelwailer marked this pull request as ready for review September 16, 2026 17:01
@johnnyelwailer
johnnyelwailer merged commit e5fbdc6 into main Sep 16, 2026
10 of 17 checks passed
@johnnyelwailer
johnnyelwailer deleted the feature/host-failed-agent-turn-swallowed-as-step-completion-8429c93c branch September 16, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant