Skip to content

fix(voice): don't deadlock a superseded AgentTask's tool or activity - #7931

Open
BlueX888 wants to merge 2 commits into
livekit:mainfrom
BlueX888:fix/agent-close-hang-update-agent-race
Open

BlueX888 wants to merge 2 commits into
livekit:mainfrom
BlueX888:fix/agent-close-hang-update-agent-race

Conversation

@BlueX888

@BlueX888 BlueX888 commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A function tool awaits an AgentTask. While it waits, update_agent() replaces the agent. When the task then completes, it unwinds on the tool task itself and the superseded-activity branch awaits old_activity.aclose() — whose tool-executor drain waits for that same non-cancellable tool task. The task awaits itself: the tool never returns, session.aclose() hangs in _wait_for_inactive, no close event is emitted, and teardown can hit RecursionError.

Reproduced on 049aea8; the new regression tests fail there (timeout in _wait_for_inactive with fut_done=True, inactive_ev=False, tool task non-cancellable; the tool-never-returns and late-unwind cases also time out).

Two adjacent patterns came out of review and are covered by this revision:

  • a tool that awaits a second AgentTask after the first was superseded: the detached close holds the activity's lock that the second task's handoff needs, so task, tool and close all hang;
  • a task that unwinds after session.aclose() returned: the close path had already walked past the superseded activity, which then leaked (tools/MCP toolsets never closed).

Fix

  • _close_detached_activity() closes a superseded activity from a detached task instead of awaiting it inside the tool task. During a session close, the close path keeps ownership; _will_close_activity() checks whether the teardown walk actually reaches the activity and the detached close is used when it does not.
  • A closed or detaching activity is never resumed again, and a second inline AgentTask on it is refused with ToolError — the same contract a tool already sees when the session is closing.
  • The session keyterm detector records its owning activity: a stale detached close no longer disables detection for the activity that now owns it.
  • drain()/aclose() are idempotent on an already closed activity.

Tests

tests/test_agent_task_close_race.py gains:

  • test_aclose_after_update_agent_while_tool_awaits_agent_task — the original close-time hang
  • test_agent_task_completion_after_update_agent_returns_to_tool — the tool that never returned after the task completed
  • test_second_agent_task_in_tool_after_update_agent_does_not_deadlock — the two-task cycle
  • test_late_agent_task_unwind_after_aclose_closes_superseded_activity — the late-unwind leak
  • test_detached_close_keeps_keyterm_detection_on_replacement — the keyterm ownership

Receipts on 049aea8: the new close-race tests fail/hang for the reasons above. With this branch: tests/test_agent_task_close_race.py 14 passed (55+ reruns), agent-session + task suites 260 passed, full unit suite 6035 passed with the same 10 pre-existing failures / 10 environment errors as 049aea8; ruff check / ruff format --check clean. Merge-tree is clean against #8147, #7956, #7930, #7940, #7957 and current main.

Residual (theoretical): nested AgentTask chains racing update_agent() could still miss a close; no reachable case was constructed.

🤖 Generated with Claude Code

@BlueX888
BlueX888 requested a review from a team as a code owner October 18, 2026 18:21
devin-ai-integration[bot]

This comment was marked as resolved.

BlueX888 and others added 2 commits October 30, 2026 06:12
An AgentTask awaited from a function tool unwinds on the tool task itself.
When update_agent() replaced the agent in the meantime, the handoff branch
awaited old_activity.aclose(), whose tool executor drain waits for that same
non-cancellable tool task, so the task awaited itself and never signalled
inactive. Session close then hung in _wait_for_inactive() and never emitted
close; if the task completed instead, the tool stayed blocked forever.

Close the superseded activity in a detached task when the session is not
closing, and let the close path own it during a close. The close path now
also drains/closes the AgentTask's own activity when update_agent() skipped
its handoff, and drain() is a no-op on an already closed activity.

Track which agent activity owns the session-scoped keyterm detector: a
detached close that lands after the replacement activity bound the detector
is stale and must not disable detection for the activity that now owns it.

Add regression tests for both the close-time hang and the tool that never
returns after the task completes, plus one for the keyterm detector staying
bound across the detached close.

Co-Authored-By: Claude Code <noreply@anthropic.com>
A tool that awaits a second AgentTask after update_agent() superseded the
first handed the tool's activity to a detached close. The second task then
resumed that activity, whose lock the detached close holds while draining
the very tool task doing the resume, so task, tool and session close all
hung. Mark the activity as owned by the detached close, refuse to start
another inline task on it, and never resume a closed or detaching activity.

A superseded AgentTask that unwinds after session.aclose() found the close
path already finished: it only walks from the session's current activity,
which update_agent() replaced, so the old activity was never closed (its
tools and MCP toolsets leaked). Track whether the in-flight close will
reach the old activity and close it from a detached task when it won't.

Add regression tests for the two-task deadlock and the late-unwind leak.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@BlueX888
BlueX888 force-pushed the fix/agent-close-hang-update-agent-race branch from 5b9bc63 to f0ca0fb Compare October 29, 2026 23:13
@BlueX888 BlueX888 changed the title fix(voice): don't close an AgentTask's activity from the tool it owns fix(voice): don't deadlock a superseded AgentTask's tool or activity Oct 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant