Skip to content

fix(runtime): retain one owner for worker process reaping - #2171

Merged
hidb4ai merged 1 commit into
masterfrom
codex/2084-worker-reaping
Oct 21, 2026
Merged

hidb4ai merged 1 commit into
masterfrom
codex/2084-worker-reaping

Conversation

@AlexStocks

Copy link
Copy Markdown
Contributor

Which issue or RFC does this PR close?

Closes #2167. Refs #2084.

Rationale for this change

Cancelling the async worker waiter does not stop its executor thread from
joining the child Process. Termination can start another join against the same
PID. On POSIX, the original thread can reap the child and pause before
publishing returncode; the competing join then cannot collect that status,
and Process.close() incorrectly reports an already reaped child as running.

This is an independently reproduced existing defect. The original #2162 CI
failure did not retain its underlying cause, so this PR does not attribute that
failure to the race or claim it was introduced by HTTP admission.

What changes are included in this PR?

  • Retain one task for the blocking Process join and share it between waiting
    and termination.
  • Shield that task from cancellation of individual async waiters, including a
    cancelled termination attempt followed by retry.
  • Wait for the join's exit-status publication before closing Process resources.
  • Add a real spawned-child regression that pauses at the POSIX
    reaping/result-publication boundary. Parameterize cancellation of the
    termination attempt; verify cleanup and an actually reaped PID.

Are there any user-facing changes?

Worker shutdown avoids an incorrect unconfirmed-exit failure in this schedule.
SIGTERM grace, SIGKILL escalation, the supervisor's five-second shutdown budget,
and errors for genuinely unconfirmed exits remain unchanged.
No API, persisted format, domain cursor, authorization, Provider or lease
ownership changes are included.

How was this change tested?

  • RED: before implementation, the controlled real-spawn regression failed with
    ValueError: Cannot close a process while it is still running.
  • WSL Ubuntu, CPython 3.11.15, frozen-lock dependencies:
    pytest tests/builtin/runtime/test_artifact_processing.py -q:
    10 passed, including both POSIX publication/cancellation cases and the
    supervisor SIGTERM-ignore / SIGKILL / stale-fence checks.
  • Native Windows, CPython 3.14.3, the same lifecycle file:
    8 passed, 2 skipped. Only the POSIX waitpid cases skip; Windows spawn,
    shutdown and stale-fence tests execute.
  • Changed-file Ruff lint/format and Ty for both Linux and Windows/Python 3.11
    targets passed. Full Linux/Python 3.11 Ty passed.
  • Full pre-commit hooks passed except native full Windows Ty, which retains the
    same 32 pre-existing POSIX/platform diagnostics in unchanged files.
  • git diff --check passed.

The regression uses real process and OS boundaries. Its short negative wait
asserts that termination stays pending while the publication gate is held; it
does not impose a subsecond startup or database success budget. Remote CI
matrix results are not claimed by these local runs.

AI usage statement

Implemented and reviewed with OpenAI Codex. Linux/Windows lifecycle results
come from actual spawned-process tests described above.

Share the blocking Process join across waiting and termination. Shield its
completion from individual waiter cancellation, preserve SIGTERM grace and
SIGKILL escalation, and close resources only after exit status is published.

Constraint: shutdown budgets and unconfirmed-exit failures remain unchanged.
Scope-risk: this fixes the reproduced POSIX race, not a proven CI root cause.
Tested: Linux real-spawn lifecycle 10 passed; Windows lifecycle 8 passed and
2 POSIX skips; Ruff lint/format; full Linux/Python 3.11 Ty; changed-file Ty
for Linux and Windows. Other pre-commit hooks pass; native full Windows Ty
retains its existing POSIX diagnostics.

Refs #2167, #2084.
Copilot AI balanced review requested due to automatic review settings October 21, 2026 10:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hidb4ai
hidb4ai merged commit 29f8ff8 into master Oct 21, 2026
44 checks passed
@hidb4ai hidb4ai added the 1.4.1 v1.4.1 release tracking label Oct 21, 2026
@PsiACE
PsiACE deleted the codex/2084-worker-reaping branch October 22, 2026 03:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.4.1 v1.4.1 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] POSIX worker shutdown can race concurrent process reaping

3 participants