Repository navigation
[awf] api-proxy: Intermittent health check failure blocks agent startup across multiple engines #2295
Description
Problem
The awf-api-proxy sidecar container intermittently fails its Docker health check during docker compose up. This blocks agent startup entirely across multiple engines (copilot, claude) and workflow types. At least 3 confirmed runs failed in the 2026-04-28 07:00–13:00 UTC window.
Affected runs:
- Sub-Issue Closer (copilot) — run 25049338576
- Daily Team Evolution Insights (claude) — run 25049605437
- Smoke CI (copilot on PR trigger) — run 25052667955
Context
Parent report: github/gh-aw#28947.
Detailed issue: github/gh-aw#28949.
This is a higher-frequency occurrence of the same pattern reported in #28898.
Root Cause
The awf-api-proxy health check in the generated docker-compose.yml (produced by src/docker-manager.ts → generateDockerCompose()) fires before the Node.js proxy process has bound its ports. The start_period is likely too short to accommodate:
- Container image cold pulls
- GitHub-hosted runner CPU contention under burst load
- Node.js module initialization latency
The containers/api-proxy/ healthcheck endpoint (likely GET /healthz or GET /) must respond within the configured timeout window, and when it doesn't, Docker marks the container unhealthy, triggering depends_on: condition: service_healthy failure in the agent service.
Proposed Solution
- Immediate (in
src/docker-manager.ts): Increase api-proxy healthcheckstart_periodto 30–45s andretriesto 5–8. This is a low-risk change that directly reduces flap frequency. - Medium-term: Add a lightweight
/healthzroute tocontainers/api-proxy/if one doesn't exist, returning200 OKas soon as the HTTP server is listening — avoiding any dependency on upstream connectivity in the health check. - Resilience: In
containers/agent/entrypoint.sh, add a pre-flight check that polls the proxy ports before proceeding, with a configurable timeout (e.g.,AWF_PROXY_STARTUP_TIMEOUT_S=60). - Observability: Log the proxy container startup time in debug mode so flap frequency can be correlated with runner load.
Generated by Firewall Issue Dispatcher · ● 436.3K · ◷
Activity
github-actions commentedon Apr 29, 2026
👋 Hello! I noticed this issue might be related to several existing issues:
docker compose uphealth check flapThese all describe the same core problem: the
awf-api-proxyhealth check failing intermittently because thestart_periodis too short for the Node.js process to bind its ports before Docker marks it unhealthy.If one of these addresses your concern, please consider closing this issue as a duplicate. Otherwise, feel free to clarify how your issue differs (e.g., new affected runs, different engine, or a regression after a previous fix)!
This is an automated message from the issue duplication detector.
lpcox commentedon Apr 29, 2026
Closing — duplicate of #2293. Already addressed by PR #2255 (merged Apr 28) which increased api-proxy health check tolerances (start_period 30s, retries 15, interval 2s, timeout 3s). Included in firewall v0.25.29+.