Repository navigation
[aw-failures] Fix CODEX_HOME variable collision breaking all Codex workflow runs (cp same-file error) #27512
Description
Problem Statement
All Codex-based workflows fail before the agent starts because CODEX_HOME is configured to /tmp/gh-aw/mcp-config/ — the same path used as the MCP config staging directory. The harness step that copies config.toml into \$\{CODEX_HOME}/ runs:
cp "/tmp/gh-aw/mcp-config/config.toml" "\$\{CODEX_HOME}/config.toml"When CODEX_HOME=/tmp/gh-aw/mcp-config, source and destination are identical, so cp exits 1 with "are the same file" and the agent job aborts before Codex ever runs (0 turns, 0 tokens).
Affected Workflows and Runs
| Workflow | PR | Run ID | Branch |
|---|---|---|---|
| Smoke Codex | #27478 | §24704312315 | copilot/bump-awf-firewall-to-v0-25-26 |
| Changeset Generator | #27478 | §24704312283 | copilot/bump-awf-firewall-to-v0-25-26 |
| Smoke Codex | #27485 | §24705019228 | copilot/update-cli-versions |
| Changeset Generator | #27485 | §24705019195 | copilot/update-cli-versions |
The bug is present on both PRs, confirming it is not a regression from either PR's changes. On PR #27478 run 24704312315, the MCP Gateway also failed to start — a secondary infra signal possibly related to firewall v0.25.26.
Root Cause
CODEX_HOME is set to the MCP config staging directory. The cp self-copy fails unconditionally.
Also observed: SECRET_CODEX_API_KEY is empty in both Changeset Generator runs — confirm the secret is provisioned correctly once the path issue is resolved.
Proposed Remediation
Set CODEX_HOME to a distinct writable path:
env:
CODEX_HOME: /tmp/gh-aw/codex-homeEnsure the directory is created before the config copy step:
mkdir -p "\$\{CODEX_HOME}"
cp "/tmp/gh-aw/mcp-config/config.toml" "\$\{CODEX_HOME}/config.toml"Success Criteria
- Smoke Codex and Changeset Generator complete
conclusion: successafter the fix - Codex agent executes ≥1 turn (currently 0 for all affected runs)
SECRET_CODEX_API_KEYconfirmed non-empty
References: §24704312315, §24705019228, §24704312283, §24705019195
Related to #27411
Generated by [aw] Failure Investigator (6h) · ● 840.5K · ◷
- expires on Apr 28, 2026, 7:29 AM UTC
Scope Escalation — 2026-04-21 13:09 UTC Investigation
The CODEX_HOME collision is confirmed on main branch for all Codex scheduled and event-triggered runs, not just PR branch runs. All 6 Codex runs in the 13:09 UTC 6-hour window failed with the identical cp same-file error:
| Run ID | Workflow | Trigger | Time |
|---|---|---|---|
| §24722769773 | AI Moderator | issue_comment |
12:38 UTC |
| §24722074102 | AI Moderator | issue_comment |
12:22 UTC |
| §24721106508 | AI Moderator | issues |
11:58 UTC |
| §24720986025 | AI Moderator | issues |
11:55 UTC |
| §24721180845 | Duplicate Code Detector | schedule |
12:00 UTC |
| §24720049478 | Daily Fact About gh-aw | schedule |
11:32 UTC |
0/6 Codex runs succeeded in this window. The fix (set CODEX_HOME=/tmp/gh-aw/codex-home) is urgently needed.
Generated by [aw] Failure Investigator (6h) · ● 329.3K · ◷
Activity