Skip to content

fix(windows): server.stop drains the config-dir ACL flight; retrying test teardown - #3257

Merged
lidge-jun merged 6 commits into
devfrom
codex/win-acl-lifecycle
Sep 2, 2026
Merged

lidge-jun merged 6 commits into
devfrom
codex/win-acl-lifecycle

Conversation

@lidge-jun

@lidge-jun lidge-jun commented Sep 2, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • server.stop() now drains the config-directory ACL harden it started. Since e5d5886 hardenConfigDir() spawns icacls.exe as a fire-and-forget flight; the child holds the directory open, Windows locking is mandatory, and anything that removed the directory after a "clean" stop got EPERM/EBUSY. That is what turned every Windows CI shard red on every branch (incl. main) since 2026-08-30. flushConfigDirHardening(dir) is a production function scoped to one directory; startServer captures its dir before loadConfig and the composite stop() awaits it in a finally so an earlier release rejection still drains the child.
  • Harness: account-store / auth-api / oauth-status move to per-test scratch homes and stub BOTH icacls runners (the sync-only stub no longer covered hardenConfigDir); 14 live-server suites remove their scratch home via removeTreeWithRetry.
  • fuck-powershell case async-child-holds-dir-after-stop (lidge-jun/fuck-powershell 56e1801) records the landmine.

Plan: devlog/_plan/260902_windows_ci_release/{000,010,011}.

Stack

PR Branch Base
1 (this) codex/win-acl-lifecycle dev
2 codex/win-fsync-fixtures codex/win-acl-lifecycle

Verification

  • tests/server-stop-config-hardening.test.ts (4): stop() pending until the held flight settles; rejected native release still drains and propagates its own error; both driven red before the fix.
  • Focused: server-stop-config-hardening, codex-account-store (43), codex-auth-api (194), 14 live-server suites — all green; bun run typecheck, bun run privacy:scan green.
  • Windows dispatch evidence lives on PR 2's head (stack tip): windows 1/4, 2/4, 3/4 SUCCESS on runs 33597649234 / 33601508392 / 33605723635 / 33610501053; 4/4 progressed from a 25-min ceiling to individual timing fixes.
  • Two read-only reviewer rounds (P2: flush skipped on rejected finalizer → fixed).

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed (devlog unit).
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults (no credential path changed; ACL hardening stays fail-closed).

Summary by CodeRabbit

  • Bug Fixes

    • Improved Windows shutdown handling so configuration-directory security operations complete before cleanup finishes.
    • Reduced intermittent EPERM and EBUSY errors when removing configuration or temporary directories after stopping the application.
    • Ensured cleanup still completes when other shutdown tasks encounter errors.
  • Tests

    • Added coverage for Windows shutdown synchronization and configuration-directory cleanup.
    • Improved temporary-directory isolation and teardown reliability across automated tests.

jun added 6 commits September 2, 2026 14:17
hardenConfigDir() spawns icacls.exe as a fire-and-forget flight (e5d5886).
The child holds the config directory open until it exits, and Windows locks
are mandatory, so removing the directory right after a clean server.stop()
returned EPERM/EBUSY — which is what every Windows CI shard has hit since
2026-08-30 in the account-store, auth-api and live-server fixtures.

flushConfigDirHardening(dir) is now a production function scoped to one
directory; startServer captures its config dir before loadConfig and the
composite stop() awaits the flight after the listeners close. Regression test
drives stop() with a held flight and proves it stays pending until release
(driven red before the fix).
…ed, retrying teardown

The fixed repo-local .tmp-codex-accounts-test meant one EPERM teardown poisoned
every later case (49/49 errors in run 33590540220 were hooks). The sync-only
icacls stub also no longer covered hardenConfigDir(), which uses the async
runner since e5d5886.
…suites; per-test homes for auth-api and oauth-status

Fourteen suites removed their scratch home with a bare rmSync right after
server.stop(). Even with stop() now draining the ACL flight, antivirus and
indexer handles on a fresh temp dir are real on the hosted image, so every
teardown goes through removeTreeWithRetry (EPERM/EBUSY/ENOTEMPTY, bounded).
codex-auth-api and oauth-status-privacy also move off fixed repo-local dirs
and stub both icacls runners, the same shape as codex-account-store.
…ase rejects

Reviewer P2: the flush sat after backgroundLifecycle/native-main release in
one finalizer, so a rejected release skipped it and the EPERM/EBUSY window
came back on the failure path. The flush now runs in finally; the original
rejection still propagates. Regression test mocks a throwing native release
and proves stop() stays pending until the flight settles (driven red).
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner September 2, 2026 09:33
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@lidge-jun

Copy link
Copy Markdown
Owner Author

Owner admin merge, user-authorized ('샤드 그린되면 ... admin 머지하고 릴리즈 force'). Head 1c41988. Evidence: Windows dispatch shards 1-3 green on four consecutive stack-tip runs (33597649234, 33601508392, 33605723635, 33610501053); Linux x4/macOS green on each; two reviewer rounds.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-02T09:40:51.941831Z 1c41988 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions github-actions Bot added the bug Something isn't working label Sep 2, 2026
@lidge-jun
lidge-jun merged commit 19b0157 into dev Sep 2, 2026
22 of 23 checks passed
@lidge-jun
lidge-jun deleted the codex/win-acl-lifecycle branch September 2, 2026 09:33
@coderabbitai

coderabbitai Bot commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Team

Run ID: d3bffd7a-1a09-441d-802a-3c9041e3c8fa

📥 Commits

Reviewing files that changed from the base of the PR and between 7d25f99 and 1c41988.

📒 Files selected for processing (23)
  • devlog/_plan/260902_windows_ci_release/000_inventory.md
  • devlog/_plan/260902_windows_ci_release/010_async_acl_lifecycle.md
  • devlog/_plan/260902_windows_ci_release/020_fsync_spill_bump.md
  • devlog/_plan/260902_windows_ci_release/030_regression_audit.md
  • devlog/_plan/260902_windows_ci_release/040_promote_and_bump.md
  • src/config/paths.ts
  • src/server/index.ts
  • tests/account-pool-management-api.test.ts
  • tests/api-usage.test.ts
  • tests/chat-completions-endpoint.test.ts
  • tests/claude-messages-endpoint.test.ts
  • tests/claude-native-passthrough.test.ts
  • tests/codex-account-store.test.ts
  • tests/codex-auth-api.test.ts
  • tests/data-plane-admission-identity.test.ts
  • tests/loopback-listener-integration.test.ts
  • tests/oauth-accounts-api.test.ts
  • tests/oauth-login-cli-live-update.test.ts
  • tests/oauth-status-privacy.test.ts
  • tests/server-kiro-completion-e2e.test.ts
  • tests/server-live.test.ts
  • tests/server-stop-config-hardening.test.ts
  • tests/vision-sidecar-e2e.test.ts

📝 Walkthrough

Walkthrough

The PR documents Windows CI failures and a remediation plan. It adds public config-directory ACL flushing, awaits that flush during server shutdown, adds lifecycle tests, and updates test cleanup to use retries and isolated temporary directories.

Changes

Windows CI release hardening

Layer / File(s) Summary
Failure inventory and remediation plan
devlog/_plan/260902_windows_ci_release/000_inventory.md, devlog/_plan/260902_windows_ci_release/010_async_acl_lifecycle.md
The plan categorizes Windows CI failures, records the asynchronous icacls.exe teardown mechanism, and defines cleanup, portability, validation, audit, and promotion work.
ACL shutdown contract
src/config/paths.ts, src/server/index.ts, tests/server-stop-config-hardening.test.ts
flushConfigDirHardening() awaits the hardening flight for a directory. startServer captures the startup config directory and awaits its flush in the stop finalizer, including when lifecycle release rejects. Tests cover pending, absent, rejected, and unrelated-directory cases.
Test teardown and isolated fixtures
tests/*
Temporary-directory cleanup uses removeTreeWithRetry. Account and OAuth tests use per-test temporary directories, stub synchronous and asynchronous ICACLS runners, flush pending hardening, restore seams, and clear environment state.
Release validation and promotion plan
devlog/_plan/260902_windows_ci_release/020_fsync_spill_bump.md, devlog/_plan/260902_windows_ci_release/030_regression_audit.md, devlog/_plan/260902_windows_ci_release/040_promote_and_bump.md
The plan specifies writable fsync handles, response-state fixture lanes, CLI path and stderr assertions, immutable-ref Windows CI verification, regression review, release promotion, and the follow-up development version bump.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant Server
  participant ACLFlush
  participant Icacls
  Caller->>Server: stop(true)
  Server->>Server: release lifecycle resources
  Server->>ACLFlush: flushConfigDirHardening(startupConfigDir)
  ACLFlush->>Icacls: await active hardening process
  Icacls-->>ACLFlush: settle
  ACLFlush-->>Server: resolve
  Server-->>Caller: stop settles
Loading

Suggested reviewers: luvs01

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/win-acl-lifecycle

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1c41988ada

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/server/index.ts
// removes the dir right after stop() settles would hit EPERM/EBUSY on Windows
// otherwise. Runs even when an earlier release rejected — that rejection still
// propagates, but not before the child is drained.
await flushConfigDirHardening(startupConfigDir);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wait for the killed ACL child to exit

When icacls.exe reaches its timeout, this await does not actually guarantee that the child has released the directory: defaultAsyncIcaclsRunner delegates to waitForSubprocessExit, whose timeout branch calls kill() and then resolves immediately without awaiting proc.exited (src/lib/bounded-subprocess.ts:12-34). Consequently, flushConfigDirHardening() can settle and server.stop() can return while the terminating child still holds the config directory, preserving the same EPERM/EBUSY race this change is intended to eliminate. Track the child-exit promise through the hardening flight and have shutdown await actual exit after issuing the kill; the new tests should also cover this timeout branch.

Useful? React with 👍 / 👎.

@lidge-jun

Copy link
Copy Markdown
Owner Author

리뷰 · 우선순위 74 / 80

이 PR은 Windows에서 서버를 끈 뒤 설정 폴더를 지우려고 할 때 나오던 EPERM/EBUSY를 고칩니다. 예전에 hardenConfigDir()가 icacls.exe를 백그라운드로만 띄워 두고, server.stop()은 그 자식 프로세스가 끝날 때까지 기다리지 않았습니다. Windows는 파일 잠금이 강제라서, 테스트가 stop 직후 임시 홈을 지우면 자주 실패했고, 2026-08-30 이후 Windows CI 샤드가 전부 빨간불이었습니다. 지금 dev HEAD(272ff6b11, 패키지 2.41.0)에는 이미 이 변경이 머지되어 있고(19b0157bb), 바로 위에는 WP2인 #3258(쓰기 가능 fsync·ACL 테스트 레인)까지 올라가 있습니다.

제품 쪽 핵심은 두 파일입니다. src/config/paths.ts에 flushConfigDirHardening(dir)를 공개해서, 그 디렉터리에 걸린 hardening flight만 await합니다. src/server/index.ts의 startServer는 loadConfig() 전에 startupConfigDir = getConfigDir()를 찍고, 합성 stop()의 finalizer에서 backgroundLifecycle.release / native release를 try한 뒤 finally에서 await flushConfigDirHardening(startupConfigDir)를 호출합니다. 리뷰어 P2 지적(앞단계 release가 터지면 flush가 스킵되던 구멍)을 finally로 막았고, 원래 거절 에러는 그대로 호출자에게 전달됩니다.

테스트 쪽은 tests/server-stop-config-hardening.test.ts 네 케이스가 계약을 고정합니다. win32 플랫폼 심을 켠 뒤 async icacls를 붙잡아 두고, 리스너가 실제로 닫힌 뒤에야 stop이 아직 pending인지 확인합니다. native release가 throw해도 flight가 끝난 뒤에야 settle되고 원본 메시지가 유지됩니다. 계정/인증/라이브 서버 여러 스위트는 고정 경로 대신 per-test scratch home을 쓰고, sync·async icacls 러너를 둘 다 스텁하며, 해제는 removeTreeWithRetry로 바꿨습니다. 계획 문서 devlog/_plan/260902_windows_ci_release/000~040도 같이 들어왔습니다.

라인 122 - tests/server-stop-config-hardening.test.ts의 거절 경로 테스트가 첫 케이스처럼 connect-refused 오라클 대신 고정 60ms sleep으로 pending을 가정합니다. 느린 러너에서는 너무 이르거나, 빠른 러너에서는 의미 없는 대기가 될 수 있습니다.
라인 2287-2291 - src/server/index.ts 시작 실패 catch 경로는 backgroundLifecycle?.releaseAfterFailedStart()와 native release만 하고 flushConfigDirHardening(startupConfigDir)를 await하지 않습니다. start가 loadConfig 이후 bind 전에 실패하면 icacls flight가 남을 수 있습니다.
경로 flushConfigDirHardening - 한 디렉터리만 비웁니다. 의도된 설계이지만, 장수 프로세스에서 홈이 바뀌고 다른 dir에 flight가 쌓이면 stop은 startup 캡처만 비웁니다(주석으로 의도 명시됨).
경로 테스트 하네스 - 제품 수정과 하네스 수정이 한 PR에 묶여 있어서 리뷰 범위는 넓지만, Windows CI 빨간불의 A/B/G 서명을 한 번에 닫는 목적에는 맞습니다.

메인테이너의 판단이 필요한 지점

  • 시작 실패 경로에도 flush를 넣을지, 아니면 테스트/운영이 stop 계약만 신뢰해도 되는지
  • 거절 경로 테스트의 60ms를 리스너-닫힘 오라클로 통일할지(후속 소형 PR vs 현 상태 유지)
  • WP2 #3258가 이미 조상이므로, 다음은 030 회귀 감사와 040 promote인지 확인

너의 추천
이미 dev에 랜딩된 상태이므로 추가 머지 작업은 없습니다. 시작 실패 flush와 60ms 오라클은 긴급이 아니면 후속 소형 이슈로 남겨 두고, Windows 릴리즈 트레인 다음 단계(030 감사 → 불변 ref Windows dispatch 증거 확인 → promote)로 진행하세요. 이 댓글은 사후 기록용입니다.

이 댓글은 grok-bot이 작성했습니다

tarunravi pushed a commit to tarunravi/opencodex that referenced this pull request Sep 14, 2026
…test teardown (lidge-jun#3257)

* devlog(windows): CI failure inventory and repair roadmap (000-040)

* devlog(windows): audit fixes — test filename, win32 platform seam in shutdown test, exact line cites

* fix(windows): server.stop drains the config-dir ACL flight it started

hardenConfigDir() spawns icacls.exe as a fire-and-forget flight (e5d5886).
The child holds the config directory open until it exits, and Windows locks
are mandatory, so removing the directory right after a clean server.stop()
returned EPERM/EBUSY — which is what every Windows CI shard has hit since
2026-08-30 in the account-store, auth-api and live-server fixtures.

flushConfigDirHardening(dir) is now a production function scoped to one
directory; startServer captures its config dir before loadConfig and the
composite stop() awaits the flight after the listeners close. Regression test
drives stop() with a held flight and proves it stays pending until release
(driven red before the fix).

* test(account-store): per-test scratch home, both icacls runners stubbed, retrying teardown

The fixed repo-local .tmp-codex-accounts-test meant one EPERM teardown poisoned
every later case (49/49 errors in run 33590540220 were hooks). The sync-only
icacls stub also no longer covered hardenConfigDir(), which uses the async
runner since e5d5886.

* test(windows): retrying scratch-home teardown across the live-server suites; per-test homes for auth-api and oauth-status

Fourteen suites removed their scratch home with a bare rmSync right after
server.stop(). Even with stop() now draining the ACL flight, antivirus and
indexer handles on a fresh temp dir are real on the hosted image, so every
teardown goes through removeTreeWithRetry (EPERM/EBUSY/ENOTEMPTY, bounded).
codex-auth-api and oauth-status-privacy also move off fixed repo-local dirs
and stub both icacls runners, the same shape as codex-account-store.

* fix(windows): drain the ACL flight even when an earlier shutdown release rejects

Reviewer P2: the flush sat after backgroundLifecycle/native-main release in
one finalizer, so a rejected release skipped it and the EPERM/EBUSY window
came back on the failure path. The flush now runs in finally; the original
rejection still propagates. Regression test mocks a throwing native release
and proves stop() stays pending until the flight settles (driven red).

---------

Co-authored-by: jun <jun@lidge.dev>
agentHits pushed a commit to agentHits/opencodex that referenced this pull request Sep 17, 2026
…test teardown (lidge-jun#3257)

* devlog(windows): CI failure inventory and repair roadmap (000-040)

* devlog(windows): audit fixes — test filename, win32 platform seam in shutdown test, exact line cites

* fix(windows): server.stop drains the config-dir ACL flight it started

hardenConfigDir() spawns icacls.exe as a fire-and-forget flight (e2f4af3).
The child holds the config directory open until it exits, and Windows locks
are mandatory, so removing the directory right after a clean server.stop()
returned EPERM/EBUSY — which is what every Windows CI shard has hit since
2026-08-30 in the account-store, auth-api and live-server fixtures.

flushConfigDirHardening(dir) is now a production function scoped to one
directory; startServer captures its config dir before loadConfig and the
composite stop() awaits the flight after the listeners close. Regression test
drives stop() with a held flight and proves it stays pending until release
(driven red before the fix).

* test(account-store): per-test scratch home, both icacls runners stubbed, retrying teardown

The fixed repo-local .tmp-codex-accounts-test meant one EPERM teardown poisoned
every later case (49/49 errors in run 33590540220 were hooks). The sync-only
icacls stub also no longer covered hardenConfigDir(), which uses the async
runner since e2f4af3.

* test(windows): retrying scratch-home teardown across the live-server suites; per-test homes for auth-api and oauth-status

Fourteen suites removed their scratch home with a bare rmSync right after
server.stop(). Even with stop() now draining the ACL flight, antivirus and
indexer handles on a fresh temp dir are real on the hosted image, so every
teardown goes through removeTreeWithRetry (EPERM/EBUSY/ENOTEMPTY, bounded).
codex-auth-api and oauth-status-privacy also move off fixed repo-local dirs
and stub both icacls runners, the same shape as codex-account-store.

* fix(windows): drain the ACL flight even when an earlier shutdown release rejects

Reviewer P2: the flush sat after backgroundLifecycle/native-main release in
one finalizer, so a rejected release skipped it and the EPERM/EBUSY window
came back on the failure path. The flush now runs in finally; the original
rejection still propagates. Regression test mocks a throwing native release
and proves stop() stays pending until the flight settles (driven red).

---------

Co-authored-by: jun <jun@lidge.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant