Skip to content

fix(worker): close resources when startup fails before run() completes - #7942

Open
BlueX888 wants to merge 2 commits into
livekit:mainfrom
BlueX888:fix/worker-startup-close-hang
Open

BlueX888 wants to merge 2 commits into
livekit:mainfrom
BlueX888:fix/worker-startup-close-hang

Conversation

@BlueX888

Copy link
Copy Markdown
Contributor

fix(worker): close resources when startup fails before run() completes

What breaks

AgentServer.run() starts the inference executor and marks the server started (_closed = False) before bringing up the HTTP/Prometheus servers, proc pool, HTTP session and API client, but only creates _close_future afterwards. If startup fails in that window (reproduced by pre-binding the default port 8081), run() raises while _closed is already False and _close_future is still None. The CLI cleanup calls aclose(), which treats _close_future is None as "nothing was started" and returns early: the inference executor's child process and its non-daemon proc_join_thread are never closed, so the CLI process cannot exit (timeout 10s => exit 124) instead of failing cleanly.

Root cause

aclose() uses _close_future is None as the "nothing was started" signal, but run() starts the first resources well before creating that future.

Fix

  • Create _close_future together with _closed = False, immediately before the first resource (the inference executor) starts, so any later startup failure runs the full shutdown path.
  • Remove the _close_future is None early return from aclose(). Every resource is already individually null-checked (inference executor, HTTP server, Prometheus server, proc pool, HTTP session, API client), so aclose() closes whatever actually started and skips only the close-future result.
  • Guard the final _close_future.set_result(None) with a None check.

Normal shutdown and drain semantics are unchanged: run() still waits on _close_future, repeated aclose() remains a no-op, and a never-started server still returns immediately.

Tests

New test tests/test_worker_connection.py::test_aclose_closes_resources_started_before_run_raises pre-binds the worker HTTP port and substitutes a fake inference executor whose start() launches a deliberately non-daemon join thread, then asserts that after run() raises, aclose() closes the executor, joins the thread, cleans up the HTTP server, and returns promptly.

On base f5ee4a8a: fails — AssertionError: aclose() must close the inference executor (assert False). With the fix: passes. Focused + neighboring files 22 passed, 2 skipped; startup/forkserver neighbors 16 passed, 5 skipped; ruff clean. The original repro with a real InferenceProcExecutor now reports inference alive: False and proc_join_thread alive: False, and the hang variant exits instead of timing out.

@BlueX888
BlueX888 requested a review from a team as a code owner October 19, 2026 05:39

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@longcw longcw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me. something nit:

Comment thread livekit-agents/livekit/agents/worker.py Outdated
logger.info("preloading plugins", extra={"packages": plugin_packages})
self._mp_ctx.set_forkserver_preload(plugin_packages)

# Startup begins here: any failure below must still run the full

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could you shorten the two new comments? The one in run() describes the old behaviour ("used to leak"); a single line such as # create before starting resources so aclose() can clean up a failed startup is enough. The one in aclose() can go, since the null checks below it say the same thing.

@BlueX888

Copy link
Copy Markdown
Contributor Author

Shortened the run() comment to the requested single line and removed the aclose() comment; no code changed. New head is b8cafb717aeaa8b53f2731937c23ccb0e80d1e6f; tests/test_worker_connection.py + tests/test_worker_config.py pass (12 passed) and ruff check is clean.

AgentServer.run() set _closed = False and started the inference executor
before creating _close_future. If a later startup step failed (e.g. the
HTTP port was already bound), the CLI's aclose() saw _close_future is
None and returned without closing the inference executor, HTTP server,
proc pool, session or API client. The executor's child process and
non-daemon join thread then kept the CLI process alive.

Create _close_future together with _closed = False, before the first
resource starts, and drop the "nothing was started" early return so
aclose() always closes the individually null-checked resources. Normal
shutdown and drain semantics are unchanged.

Add a regression test that pre-binds the worker's HTTP port, then
asserts aclose() closed the (fake) inference executor and its
non-daemon thread and cleaned up the HTTP server.
@BlueX888
BlueX888 force-pushed the fix/worker-startup-close-hang branch from b8cafb7 to 9b3df53 Compare October 26, 2026 23:56

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants