Repository navigation
[awf] agent+api-proxy: P0/P1 failure cluster — api-proxy health check + node not found in copilot container #2294
Description
Problem
This is a combined P0/P1 failure report covering 28 failed/cancelled runs out of 40 total over the 07:00–13:00 UTC window on 2026-04-28. Two distinct failure clusters were identified:
P0 — awf-api-proxy health check timeout (at least 3 confirmed runs): Affects Sub-Issue Closer (copilot), Daily Team Evolution Insights (claude), Smoke CI (copilot). Container starts but health check times out, blocking docker compose up entirely.
P1 — node: command not found in copilot agent container: At least 2 scheduled copilot-engine workflows fail inside the container because node is not on PATH. Affects: Daily Issues Report Generator, Daily News.
Context
Parent investigation: github/gh-aw#28947. The api-proxy health check failure is also separately tracked.
Root Cause
P0 (api-proxy health check): See separate tracking issue for #28898/#28949 — tight healthcheck timing under runner load.
P1 (node not found): The containers/agent/entrypoint.sh chroots to /host and drops capabilities before running the user command. The PATH inside the chroot may not include the directory where the host's node binary lives (e.g., /usr/local/bin/node, /opt/hostedtoolcache/node/.../bin). The bind mounts include /usr, /bin, /sbin read-only, but /usr/local or nvm/volta paths under /opt may not be fully covered.
Proposed Solution
P1 fix:
- In
containers/agent/entrypoint.sh: After chroot, constructPATHby including common Node.js installation locations:/usr/local/bin,/opt/hostedtoolcache/node/*/bin(glob expanded),/home/runner/.nvm/versions/node/*/bin. - Alternatively: Add a
--extra-path <dir>CLI flag insrc/cli.tsthat appends additional directories to the agent containerPATH. - In
src/docker-manager.ts: When generating the Docker Compose env section, include the host'sPATHvalue or a curated superset to ensure tool availability inside the chroot.
P0 fix: Tracked in the api-proxy health check issues.
Generated by Firewall Issue Dispatcher · ● 436.3K · ◷
Activity
github-actions commentedon Apr 29, 2026
👋 Hello! I noticed this issue might be related to existing issues:
P0 — api-proxy health check:
P1 — node not found in copilot container:
node: command not foundon runner bootstrap #2159 - Copilot CLI fails withnode: command not foundon runner bootstrapIn particular, #2275 appears to be a near-identical combined P0/P1 report. Please review the linked issues to see if any already address your concern, or clarify how this report differs (e.g., different time window, affected runs, or root cause findings).
This is an automated message from the issue duplication detector.
lpcox commentedon Apr 29, 2026
Closing — both failure clusters already addressed by merged PRs:
P0 (api-proxy health check timeout): Fixed by PR #2255 (merged Apr 28) — increased health check tolerances (start_period 30s, retries 15, interval 2s, timeout 3s).
P1 (node: command not found): Fixed by PR #2160 (merged) — added
~/.nvmto agent bind mounts andAWF_REQUIRE_NODEpreflight check for Copilot workflows, ensuring Node.js is discoverable inside the chroot.Both fixes are included in firewall v0.25.29+.