[awf] api-proxy: GOOGLE_API_KEY invalid — all Smoke Gemini runs failing since 2026-04-29 #2296

Problem

All Smoke Gemini workflow runs have been failing since 2026-04-29 00:21 UTC with HTTP 400 INVALID_ARGUMENT / API_KEY_INVALID from generativelanguage.googleapis.com. The agent makes zero LLM calls, zero tool calls, and produces no output before terminating. This is a P0 blocker for all Gemini-based agentic workflows.

Context

Reported in github/gh-aw#29030.

Affected runs (sample):

• Run 25084493525 — branch copilot/align-docs-with-canonical-apm — 2026-04-29 00:21
• Run 25085302973

Failure message:

HTTP 400 INVALID_ARGUMENT
reason: API_KEY_INVALID
service: generativelanguage.googleapis.com
message: "API key not valid. Please pass a valid API key."

Root Cause

The GOOGLE_API_KEY repository secret has expired or been rotated/revoked. This is an operational credential issue, not a code bug. However, AWF's Gemini proxy path may also be contributing: when --enable-api-proxy is active, the sidecar injects the key — if the sidecar has a bug in Gemini key injection (see also #25944 tracking issue), a valid-looking key could still be forwarded incorrectly.

Two possible root causes to rule out in order:

1. Secret expired — GOOGLE_API_KEY in the github/gh-aw Actions secrets is invalid/revoked
2. Proxy injection bug — the api-proxy sidecar doesn't correctly inject GOOGLE_API_KEY as a ?key= query parameter for Gemini requests

Proposed Solution

1. Immediate operational fix: Rotate the GOOGLE_API_KEY secret in the github/gh-aw repository Actions settings.
2. Verify proxy injection: After rotation, if failures persist, audit containers/api-proxy/ Gemini route handler to confirm the key is injected as ?key=<value> query parameter (not as Authorization: Bearer).
3. Add secret expiry alerting: Add a scheduled workflow that validates API keys with a cheap probe call and creates an issue if any key returns API_KEY_INVALID.
4. Add AWF startup validation: In the api-proxy sidecar startup, probe each configured API key and log a warning (not failure) if a key appears invalid, so the issue is caught before the agent runs.

Generated by Firewall Issue Dispatcher · ● 436.3K · ◷