[awf] api-proxy: GOOGLE_API_KEY invalid — Smoke Gemini runs failing with API_KEY_INVALID since 2026-04-29 #2288

Problem

All Smoke Gemini workflow runs have been failing since at least 2026-04-29 00:21 UTC with HTTP 400 INVALID_ARGUMENT / API_KEY_INVALID from generativelanguage.googleapis.com. The agent makes zero LLM calls before terminating. This is a P0 blocker for all Gemini-based smoke tests.

HTTP 400 INVALID_ARGUMENT
reason: API_KEY_INVALID
service: generativelanguage.googleapis.com
message: "API key not valid. Please pass a valid API key."

Context

• Original issue: [aw-failures] Fix P0: Gemini API key (GOOGLE_API_KEY) is invalid — all Smoke Gemini runs failing since 2026-04-29 gh-aw#29030
• Affected runs: §25084493525, §25085302973
• No successful Smoke Gemini runs in the last 24h; last success was 2026-04-14 or earlier
• Related prior issue: gh-aw#25944 (Gemini API key rejected by proxy sidecar)

Root Cause

The GOOGLE_API_KEY repository secret in github/gh-aw is invalid (expired, deleted, or replaced with an incorrect value). The API proxy sidecar (containers/api-proxy/) injects this key for Gemini requests, but there is no startup health check for Gemini — the sidecar only validates ANTHROPIC_BASE_URL, OPENAI_BASE_URL, and COPILOT_API_URL on startup. The bad key is therefore not caught until the first real LLM call.

Proposed Solution

Immediate (Ops)

1. Navigate to Settings → Secrets and variables → Actions in github/gh-aw
2. Rotate/replace GOOGLE_API_KEY with a valid Gemini API key (project must have generativelanguage.googleapis.com enabled)
3. Re-run a Smoke Gemini job to confirm recovery

Code Fix (Firewall)

Add a Gemini startup health check in the API proxy sidecar, analogous to the existing checks for OpenAI/Anthropic/Copilot. This prevents silent failures that only surface at first LLM call:

• In containers/api-proxy/, add a GEMINI_API_KEY presence check during container startup (similar to how ANTHROPIC_API_KEY is validated before forwarding)
• Emit a clear [WARN] GEMINI_API_KEY not set or empty log line at sidecar startup so the health check in docker-compose.yml can detect misconfiguration early

Success Criteria

• Smoke Gemini run completes with conclusion: success
• Firewall logs show generativelanguage.googleapis.com:443 receiving successful responses
• Agent produces at least 1 LLM turn and valid safe output

Generated by Firewall Issue Dispatcher · ● 986.3K · ◷