Repository navigation
Conversation
longcw
approved these changes
Oct 20, 2026
longcw
left a comment
Contributor
There was a problem hiding this comment.
looks good to me. something nit:
| logger.info("preloading plugins", extra={"packages": plugin_packages}) | ||
| self._mp_ctx.set_forkserver_preload(plugin_packages) | ||
|
|
||
| # Startup begins here: any failure below must still run the full |
Contributor
There was a problem hiding this comment.
nit: could you shorten the two new comments? The one in run() describes the old behaviour ("used to leak"); a single line such as # create before starting resources so aclose() can clean up a failed startup is enough. The one in aclose() can go, since the null checks below it say the same thing.
Contributor
Author
|
Shortened the |
AgentServer.run() set _closed = False and started the inference executor before creating _close_future. If a later startup step failed (e.g. the HTTP port was already bound), the CLI's aclose() saw _close_future is None and returned without closing the inference executor, HTTP server, proc pool, session or API client. The executor's child process and non-daemon join thread then kept the CLI process alive. Create _close_future together with _closed = False, before the first resource starts, and drop the "nothing was started" early return so aclose() always closes the individually null-checked resources. Normal shutdown and drain semantics are unchanged. Add a regression test that pre-binds the worker's HTTP port, then asserts aclose() closed the (fake) inference executor and its non-daemon thread and cleaned up the HTTP server.
BlueX888
force-pushed
the
fix/worker-startup-close-hang
branch
from
October 26, 2026 23:56
b8cafb7 to
9b3df53
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(worker): close resources when startup fails before run() completes
What breaks
AgentServer.run()starts the inference executor and marks the server started (_closed = False) before bringing up the HTTP/Prometheus servers, proc pool, HTTP session and API client, but only creates_close_futureafterwards. If startup fails in that window (reproduced by pre-binding the default port 8081),run()raises while_closedis alreadyFalseand_close_futureis stillNone. The CLI cleanup callsaclose(), which treats_close_future is Noneas "nothing was started" and returns early: the inference executor's child process and its non-daemonproc_join_threadare never closed, so the CLI process cannot exit (timeout 10s=> exit 124) instead of failing cleanly.Root cause
aclose()uses_close_future is Noneas the "nothing was started" signal, butrun()starts the first resources well before creating that future.Fix
_close_futuretogether with_closed = False, immediately before the first resource (the inference executor) starts, so any later startup failure runs the full shutdown path._close_future is Noneearly return fromaclose(). Every resource is already individually null-checked (inference executor, HTTP server, Prometheus server, proc pool, HTTP session, API client), soaclose()closes whatever actually started and skips only the close-future result._close_future.set_result(None)with aNonecheck.Normal shutdown and drain semantics are unchanged:
run()still waits on_close_future, repeatedaclose()remains a no-op, and a never-started server still returns immediately.Tests
New test
tests/test_worker_connection.py::test_aclose_closes_resources_started_before_run_raisespre-binds the worker HTTP port and substitutes a fake inference executor whosestart()launches a deliberately non-daemon join thread, then asserts that afterrun()raises,aclose()closes the executor, joins the thread, cleans up the HTTP server, and returns promptly.On base
f5ee4a8a: fails —AssertionError: aclose() must close the inference executor(assert False). With the fix: passes. Focused + neighboring files 22 passed, 2 skipped; startup/forkserver neighbors 16 passed, 5 skipped; ruff clean. The original repro with a realInferenceProcExecutornow reportsinference alive: Falseandproc_join_thread alive: False, and the hang variant exits instead of timing out.