Repository navigation
[aw-failures] Failure Investigation Report — 2026-04-21 13:09 UTC (6h window) #27575
Description
Executive Summary
Investigation of the 13:09 UTC 6-hour window identified 10 failed runs out of 45 total (22% failure rate). The dominant failure is the CODEX_HOME variable collision bug (#27512) — now confirmed to affect all Codex scheduled and event-triggered runs on main, a significant scope escalation from the original PR-only evidence. All 6 Codex runs in this window failed with the same cp: same file error before the agent could start. Additionally, 4 Copilot CLI workflows failed with workflow-specific patterns (timeout, 27-turn agent exit). No Design Decision Gate runs were observed in this window, so #27513 status remains unverified.
Failure Clusters
| # | Cluster | Engine | Runs | Root Cause | Priority | Tracking |
|---|---|---|---|---|---|---|
| 1 | CODEX_HOME collision — all Codex workflows | Codex v0.121.0 | 6 | CODEX_HOME=/tmp/gh-aw/mcp-config causes cp same-file error, aborting agent before any turn |
P0 | #27512 (scope expanded) |
| 2 | Terminal Stylist timeout | Copilot CLI | 1 | Copilot CLI action hit 10-min hard timeout at turn 13 | P1 | #27560 |
| 3 | Daily Firewall Logs 27-turn failure | Copilot CLI | 1 | Agent completed 27 turns then failed | P2 | #27571 |
| 4 | Daily Issues Report Generator startup failure | Copilot CLI | 1 | Agent job failed at 0 turns (detection+safe_outputs skipped) | P2 | #27534 |
| 5 | Daily Community Attribution Updater | Copilot CLI | 1 | Agent error, not yet deep-audited | P2 | #27546 |
| 6 | Multi-Device Docs Tester | Claude Code | 1 | missing_tool_count=1 — called a missing tool |
P2 | — |
Evidence
Cluster 1 — CODEX_HOME collision (P0)
Confirmed from run §24722769773 (AI Moderator, Codex, 12:38 UTC):
cp: '/tmp/gh-aw/mcp-config/config.toml' and '/tmp/gh-aw/mcp-config/config.toml' are the same file
##[error]Process completed with exit code 1.
The MCP Gateway starts successfully (Gateway started with PID, 30 tools registered), then the harness copies config.toml into $CODEX_HOME. Because CODEX_HOME=/tmp/gh-aw/mcp-config, src and dst are identical → cp exits 1 → agent step aborts with 0 turns.
Scope expansion: Issue #27512 originally documented this on Smoke Codex and Changeset Generator triggered by PR events. Now confirmed on:
- AI Moderator × 4 (12:38, 12:22, 11:58, 11:55 UTC) —
issue_commentandissuesevent triggers - Duplicate Code Detector × 1 (12:00 UTC) — schedule trigger
- Daily Fact About gh-aw × 1 (11:32 UTC) — schedule trigger
All 6/6 Codex runs in the window failed; 0 succeeded. The bug is not limited to PR branches — it is present on main for all Codex workflow runs.
Cluster 2 — Terminal Stylist timeout (P1)
Run §24721200524: 13 turns × 744k tokens in 13.3 min total. The Copilot CLI action step hit its 10-minute hard wall:
##[error]The action 'Execute GitHub Copilot CLI' has timed out after 10 minutes.
The agent had produced a create_discussion safe-output before the timeout, so partial work was delivered. Cause: Terminal Stylist is exploration-heavy (code analysis across many files), consistently consuming >10 min of agent time.
Cluster 4 — Daily Issues Report Generator pre-agent failure
Run §24717859123: 0 turns, 6.1 min duration, detection/safe_outputs/upload_assets all skipped. This pattern (hard abort before first turn) on a Copilot CLI workflow warrants investigation — it is distinct from the timeout pattern. Possible causes: Docker startup issue, credential/secret unavailability, or a pre-agent harness step failure.
Existing Issue Correlation
| Issue | Status | Finding |
|---|---|---|
| #27512 — CODEX_HOME collision | Open | Bug confirmed unresolved; scope escalated to all Codex runs on main |
| #27513 — Design Decision Gate max-turns | Open | No DDG runs in this 6h window — unverifiable, treat as still open |
| #27412 — AI Moderator Codex chatgpt.com firewall | Open | Current AI Moderator failures are CODEX_HOME (cp error), not chatgpt.com block; #27412 cannot be confirmed resolved until CODEX_HOME is fixed |
| #27469 — Prior Failure Report (01:21 UTC today) | Open | Different window; DDG max-turns root cause; this report is the follow-on |
| #27411 — Issue Group | Open | Remains the parent for new sub-issues |
Fix Roadmap
| Priority | Fix | Owner | Effort |
|---|---|---|---|
| P0 | Set CODEX_HOME=/tmp/gh-aw/codex-home (distinct from MCP config dir) in Codex engine harness; mkdir -p before cp |
per #27512 | Low |
| P1 | Raise Terminal Stylist Copilot CLI timeout beyond 10 min, or split analysis into deterministic pre-steps | — | Low |
| P2 | Investigate Daily Issues Report Generator pre-agent hard failure (0 turns, Copilot CLI) | — | Medium |
| P2 | Once CODEX_HOME fixed, re-run AI Moderator to verify chatgpt.com block (#27412) is separate or was masked | — | Low |
Sub-Issues
No new sub-issues created in this run — the root causes map to existing open issues (#27512, #27513, #27412). #27512 updated with expanded-scope evidence.
References:
- §24722769773 (AI Moderator Codex — cp error confirmed)
- §24721200524 (Terminal Stylist — timeout)
- §24717859123 (Daily Issues Report Generator — pre-agent failure)
Generated by [aw] Failure Investigator (6h) · ● 329.3K · ◷
- expires on Apr 28, 2026, 1:20 PM UTC
Activity