[BlueX888]

What breaks

When the worker fails after startup — missing or bad credentials, an
unreachable LiveKit server, invalid config — the CLI logs the error but the
process exits with code 0. (A failure before the internals initialize,
e.g. no registered rtc_session entrypoint, instead ends in a masking
AttributeError, described below.)

$ unset LIVEKIT_URL LIVEKIT_API_KEY LIVEKIT_API_SECRET
$ python -m livekit.agents start myagent.py
{"message": "worker failed", "level": "ERROR", ...ValueError: ws_url is required...}
$ echo $?
0

The same happens for dev and for the deprecated rich CLI (start/dev).
Orchestrators watch the exit code, so they see a clean stop: systemd reports
a successful unit, docker/k8s probes treat the container as exited normally,
and CI passes a job whose worker never came up. Nothing alerts, nothing
restarts.

A second, adjacent problem: when run() fails before the server internals
are initialized (e.g. no rtc_session registered), the subsequent
server.drain() raises AttributeError: 'AgentServer' object has no attribute '_proc_pool'. That secondary error escapes the CLI and becomes the final
traceback, masking the real failure that had just been logged.

Root cause

_run_worker in livekit/agents/cli/cli.py wrapped server.run() in a bare
except Exception that only logged the failure:

async def _worker_run(worker: AgentServer) -> None:
    try:
        await server.run(devmode=devmode, unregistered=False)
    except Exception:
        logger.exception("worker failed")

The worker task then completed "successfully", the normal drain/aclose
shutdown ran, _run_worker returned, and main() (python -m livekit.agents) finished with return 0. The startup failure never reached
the exit code. In the early-failure case, drain() at the same site raised
AttributeError on the uninitialized internals, which then replaced the
original error in the final traceback.

The fix

Confined to _run_worker (livekit/agents/cli/cli.py):

• _worker_run records the failure in addition to logging it.
• After the usual drain/aclose shutdown and loop cleanup, the CLI raises
  SystemExit(1) when the worker failed, so every entry point
  (python -m livekit.agents start, the deprecated rich CLI start/dev,
  and jupyter) reports the failure: the CLI processes exit non-zero; in a
  notebook IPython surfaces it as a cell error. The graceful-shutdown path
  still runs first.
• A secondary exception from drain()/aclose() after a failed run is
  logged as a warning instead of escaping and masking the original error.
  When the worker did not fail, such exceptions still propagate exactly
  as before (the behavior pinned by
  test_drain_timeout.py::test_aclose_skipped_on_non_exitcli_non_timeout_exception).

Unchanged on purpose:

• A normal signal shutdown (SIGTERM/SIGINT) still exits 0.
• run() raising SystemExit/CancelledError (BaseException) is not caught
  by the except Exception, so signal-driven exits keep their current
  semantics.

Test evidence

New regression tests in tests/test_cli_exit.py, following the file's
existing stub-server pattern:

• test_worker_startup_failure_exits_nonzero (parametrized over start and
  dev): a server whose run() raises at startup must produce a non-zero
  SystemExit, with cleanup still invoked (aclose; drain is skipped in
  the dev variant by pre-existing devmode logic). Failed before the fix
  (_run_worker returned normally — process exit 0), passes after.
• test_worker_startup_failure_before_init_is_not_masked: a server that
  fails before its internals exist, with drain()/aclose() raising
  AttributeError like the real AgentServer. The exit must be a non-zero
  SystemExit (not the escaping AttributeError) and the original
  "worker failed" error must remain the one logged. Failed before the fix
  (AttributeError escaped), passes after.

The full tests/test_cli_exit.py file passes; tests/test_cli_log_level.py,
tests/test_drain_timeout.py and tests/test_worker_config.py (the other
suites covering the _run_worker call boundary) show no regressions.

[BlueX888]

Thanks for the review — assessed against worktree head 7b21edadc (base f5481499). TL;DR: the factual core (aclose() not called when drain() raises a non-timeout error after the worker failed) is correct, but it is pre-existing behavior, not introduced by this PR, and the claimed harm — job processes outliving the worker — does not occur. Declining the recommended fix in this PR; evidence below.

1. The aclose() skip is pre-existing. At head, a non-timeout drain error after worker_failed is downgraded to a warning and aclose() is indeed not reached in that block. But at base the same block caught only _ExitCli, so a non-timeout drain error propagated straight out — skipping aclose() identically and escaping _run_worker, masking the original worker failure (independently reproduced at base in our review notes). The only behavior this PR changes in that path is which error is reported; the child-governing cleanup (the unchanged finally block) is byte-identical to base. The skip is also pinned upstream (tests/test_drain_timeout.py:140-159 asserts aclose is not awaited on a non-timeout drain error when the worker did not fail); our conditional downgrade preserves it.

2. Children do not survive — three independent guarantees.

• Cancelling those tasks IS the pool's shutdown. ProcPool.aclose() is literally "cancel _main_task and wait" (ipc/proc_pool.py:107-112), and _main_task's CancelledError handler runs each executor's aclose() (proc_pool.py:299-303) — graceful ShutdownRequest, ack wait, then kill (supervised_proc.py:309-351). _run_worker's unchanged finally cancels those same tasks, so the per-executor close runs even when AgentServer.aclose() was skipped. Verified empirically through the real _run_worker at head with a real ProcPool holding two executors with running jobs, run() failing late, drain() raising a non-timeout error (your exact scenario): exit 1, server.calls == ['drain'], aclose skipped, both executors closed by the finally-block cancel, leaked children: NONE.
• Children self-terminate on parent loss by design. They ignore SIGINT/SIGTERM (job_proc_lazy_main.py:5-10) but exit when the parent's duplex EOFs (proc_client.py:108-113, 142-157), with a ~70s ping backstop.
• The realistic children-alive case still runs aclose(). The common drain failure with running jobs is the timeout, which is caught by the inner handler regardless of worker_failed (cli.py:442-445) — aclose runs. A non-timeout drain raise with children alive requires an exception from proc.join() (that executor already ended) or AttributeError('_proc_pool') (pool never existed) — the harm scenario doesn't attach to any reachable state.

3. Why decline here. The recommended fix wouldn't change child fate (see 2a; where the pool's _main_task already died, ProcPool.aclose() on a done task is a no-op anyway); re-staging the shutdown sequence is a semantics redesign that belongs in its own PR. For completeness, the hardening is feasible if wanted: AgentServer.aclose() is idempotent and explicitly tolerates a failed run() (worker.py:1027-1039), so a best-effort aclose in the worker_failed branch would be safe — happy to review a dedicated upstream PR for that.

[BlueX888]

Follow-up: close the worker when drain() fails after run() failed

Updated in 3c764d64b on this branch.

What was still wrong

When run() failed and drain() then raised a non-timeout error, _run_worker
downgraded the drain error to a warning but skipped server.aclose(), because
drain() and aclose() shared one try block. AgentServer.aclose() is the
only teardown path for the inference executor, HTTP session/server, Prometheus
server, API client and message channel (worker.py:1056-1071), so that skip
left those resources without their explicit close while the CLI was already
exiting non-zero.

Correction to the earlier note in this thread: the proc pool and child job
processes are not part of this claim. The unchanged finally block cancels the
loop tasks, and ProcPool._main_task closes its executors on CancelledError,
so this patch does not argue that children would outlive the worker. The
regression test pins only that aclose() is still called explicitly.

Change

_run_worker now records whether aclose() was attempted. If the shutdown
block fails after a worker failure and aclose() was not reached, it calls
aclose() in a separate stage. aclose() is idempotent and explicitly
tolerates a failed run(): it closes whatever components were started and,
when the server is already closed, awaits _close_future under
contextlib.suppress(Exception) so a stored run() error cannot escape. An
error from that stage is logged as a warning instead of escaping, and in the
partially-initialized case it stays a no-op.

Unchanged: when the worker did not fail, a non-timeout drain() error still
propagates and still skips aclose() (pinned by
tests/test_drain_timeout.py::TestDrainTimeout::test_aclose_skipped_on_non_exitcli_non_timeout_exception).
_ExitCli handling and the drain-timeout path are untouched.

Regression test

tests/test_cli_exit.py::test_worker_failure_still_closes_server_when_drain_raises
uses a stub server whose run() and drain() both raise. It asserts that
aclose() is still called explicitly; it does not claim to observe process
lifetime. On 7b21edadc it fails with assert ['drain'] == ['drain', 'aclose'];
with the fix it passes, the drain error stays a warning, and the exit is still
non-zero.

Verification (head 3c764d64b)

• uv run pytest tests/test_cli_exit.py -q: 9 passed
• CLI/worker/proc suites (test_cli_console_output, test_cli_exit,
  test_cli_log_level, test_cli_logging_encoding, test_drain_timeout,
  test_supervised_proc_memory, test_worker_config, test_worker_connection):
  70 passed, 1 skipped
• ruff format --check and ruff check on the changed files: clean
• uv run --group typing mypy --untyped-calls-exclude=smithy_aws_core --platform linux -p livekit.agents:
  no issues in 229 source files