[aw-failures] Daily Model Inventory Checker: Copilot CLI silent startup crash regression (matches closed #30043) #32245

Summary

The Daily Model Inventory Checker scheduled workflow has failed two consecutive nightly runs (2026-05-14T00:14Z and 2026-05-15T00:11Z). The agent job exits with code 1 within ~1 second of spawning the copilot CLI binary, producing zero stdout and zero stderr — a silent startup crash. The fingerprint is identical to previously-closed #30043, suggesting that fix has regressed or was incomplete.

Affected workflow

• Daily Model Inventory Checker (scheduled, main, engine: copilot)
• Last 10 runs: 8 success / 2 failure — both failures are the two most recent runs.
• Last successful run: §25704992041 on 2026-05-12T00:11Z.

Failed runs analyzed

Probable root cause

From agent-stdio.log of run §25892978215:

[copilot-harness] starting: command=/usr/local/bin/copilot maxRetries=3 ...
[copilot-harness] pre-flight: command is accessible and executable: /usr/local/bin/copilot
[copilot-harness] resolved --prompt-file: path=/tmp/gh-aw/aw-prompts/prompt.txt size=19500B
[copilot-harness] awf-reflect: fetched 25 model(s) from (apiproxy/redacted)
[copilot-harness] attempt 1: spawning: /usr/local/bin/copilot --add-dir /tmp/gh-aw/ --log-level all ...
[copilot-harness] attempt 1: process started (pid=178)
[copilot-harness] attempt 1: process exit event exitCode=1
[copilot-harness] attempt 1: process closed exitCode=1 duration=1s stdout=0B stderr=0B hasOutput=false
[copilot-harness] attempt 1: no output produced — not retrying (possible causes: binary not found, permission denied, auth failure, or silent startup crash)
[copilot-harness] done: exitCode=1 totalDuration=1s

Key signals:

• Pre-flight passes: the binary is present, executable, and the prompt file is resolved.
• Network is healthy: awf-reflect successfully fetches 25 models from the api-proxy before spawn.
• Process spawns (pid=178), exits 1 second later with exitCode=1, stdout=0B, stderr=0B.
• Harness correctly declines to retry (silent crash is not transient).
• agent_output.json is {"items":[]} — no work was performed.

Audit-diff vs last successful run

run_metrics_diff:
  run1 (failed):   token_usage=0,  duration=2m21s,  total_requests=0
  run2 (success):  token_usage=21,226,834 effective, duration=8m15s, total_requests=30
firewall_diff:
  api.githubcopilot.com:443 — run1 allowed=2, run2 allowed=32 (volume_change=+1500%)
  no new/removed domains, no blocked requests, no anomalies
mcp_tools_diff:
  run1: no safeoutputs tool calls
  run2: safeoutputs.create_issue called once

Historical context

Closed issue #30043 — "Daily Model Inventory Checker: Copilot CLI silent startup crash (exit code 1)" — has the same fingerprint (silent exit, 0 bytes output, ~1 second runtime). Whatever mitigation landed there appears to have regressed. Without diagnostic output from the binary itself, possible causes (per the harness message) are:

1. Auth / credential isolation issue specific to this workflow's --allow-tool set
2. A change in /usr/local/bin/copilot (version bump, binary corruption, missing shared lib)
3. A prompt-file or argv shape that the current CLI rejects without producing stderr
4. Sandbox / chroot interaction with a newer copilot binary

The argv passed to copilot for this workflow is unusually long (~30+ --allow-tool shell(...) flags) — see agent-stdio.log lines 6160–6184 in run §25892978215. If the CLI has a new argv length limit, this workflow could be the first to hit it.

Proposed remediation

1. Reproduce locally with the exact argv captured from the failing run (agent-stdio.log of run §25892978215) and the same /usr/local/bin/copilot version pinned in the runner image. Confirm whether the binary itself crashes silently or whether stdio is being swallowed by the harness/chroot.
2. Capture stderr unconditionally. The harness reports stderr=0B — verify that the spawn is not redirecting/dropping stderr in copilot_harness.cjs. If stderr is genuinely empty, add a strace -f -e trace=execve,exit_group -o /tmp/... wrapper for the first retry attempt to surface the failure point.
3. Bisect argv complexity. Try the same workflow with a minimal --allow-tool set to determine whether argv length / shell-pattern parsing is the trigger. If so, refactor this workflow's allowed-tool list (many shell(jq . /tmp/...) and shell(cat /tmp/...) entries could be collapsed).
4. Pin copilot CLI version in the runner image if a newer release introduced the regression.

Success criteria

• Next scheduled run of Daily Model Inventory Checker produces agent_output.json with non-empty items and conclusion: success.
• Harness produces non-empty stdout/stderr from the copilot process, so future silent-crash regressions are diagnosable from logs alone.
• Two consecutive successful nightly runs to confirm the fix is not flaky.

References:

• Failed run: (https://github.com/github/gh-aw/actions/runs/25892978215)
• Prior-night failure (context): (https://github.com/github/gh-aw/actions/runs/25833891456)
• Last success baseline: (https://github.com/github/gh-aw/actions/runs/25704992041)

Generated by 🔍 [aw] Failure Investigator (6h) · ● 15.4M · ◷

• expires on May 22, 2026, 1:36 AM UTC