Repository navigation
[aw-failures] Fix P0: Gemini API key (GOOGLE_API_KEY) is invalid — all Smoke Gemini runs failing since 2026-04-29 #29030
Description
Problem Statement
The GOOGLE_API_KEY secret used by the Smoke Gemini workflow is invalid or expired. All Smoke Gemini runs since at least 2026-04-29 00:21 UTC are failing immediately on first LLM call with:
HTTP 400 INVALID_ARGUMENT
reason: API_KEY_INVALID
service: generativelanguage.googleapis.com
message: "API key not valid. Please pass a valid API key."
The agent makes zero LLM calls, zero tool calls, and produces no output before terminating.
Affected Workflows and Runs
| Run | Branch | Time (UTC) | Status |
|---|---|---|---|
| §25084493525 | copilot/align-docs-with-canonical-apm |
2026-04-29 00:21 | failure — API_KEY_INVALID |
| §25085302973 | copilot/align-docs-with-canonical-apm |
2026-04-29 00:49 | failure — API_KEY_INVALID |
No successful Smoke Gemini run found in the last 24h. Prior successful runs date to 2026-04-14 or earlier.
Root Cause
The GOOGLE_API_KEY repository secret is invalid. Evidence from Gemini CLI error dump files (gemini-client-error-*.json) on run 25085302973 confirms the identical error at two call sites:
Models.generateContent→NumericalClassifierStrategy.routeModels.generateContentStream→Turn.run
The key is either expired, deleted, or replaced with an incorrect value.
Proposed Remediation
- Go to Settings → Secrets and variables → Actions for
github/gh-aw - Locate
GOOGLE_API_KEY - Rotate or replace it with a valid Gemini API key (project must have
generativelanguage.googleapis.comenabled) - Re-run a Smoke Gemini job to verify recovery
Success Criteria
- Smoke Gemini run completes with
conclusion: success - Firewall logs show
generativelanguage.googleapis.com:443receiving successful responses (no blocked/error domains) - Agent produces at least 1 LLM turn and a
noopor valid safe output
References
- Parent group issue: [aw-failures] [aw] Failure Investigator (6h) - Issue Group #28948
- Existing failure issue: [aw] Smoke Gemini failed #29014 (Smoke Gemini failed — contains raw error traceback)
- Audit analysis: Investigator run §25086158628
References:
Note
🔒 Integrity filter blocked 6 items
The following items were blocked because they don't meet the GitHub integrity level.
- #25944
search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - #23655
search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - Add
<img>to safe-outputs HTML tag allowlist #29009list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - Flaky awf-api-proxy unhealthy failure in downstream Copilot workflow #28898
list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - MCP gateway fails on ARC self-hosted runners with dind sidecar — "Invalid container ID format" + "Docker socket not found" #28888
list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - Support self-hosted runners whose service account/home is not /home/runner #27260
list_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
To allow these resources, lower min-integrity in your GitHub frontmatter:
tools:
github:
min-integrity: approved # merged | approved | unapproved | noneGenerated by [aw] Failure Investigator (6h) · ● 668.8K · ◷
- expires on May 6, 2026, 1:35 AM UTC
Activity
github-actions commentedon Apr 29, 2026
🍪 *Issue Monster selected this for Copilotecho BEGIN___COMMAND_OUTPUT_MARKER ; PS1= ; PS2= ; unset HISTFILE ; EC=0 ; echo ___BEGIN___COMMAND_DONE_MARKER___0 ; }
I've identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.
If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.
Om nom nom! 🍪
Caution
Security scanning requires review for Issue Monster
Details
Potential security threats were detected in the agent output. The workflow output should be reviewed before merging.
Review the workflow run logs for details.
lpcox commentedon Apr 29, 2026
🔗 AWF tracking issue: https://github.com/github/gh-aw-firewall/issues/github/gh-aw-firewall#2288
lpcox commentedon Apr 29, 2026
🔗 AWF tracking issue: https://github.com/github/gh-aw-firewall/issues/github/gh-aw-firewall#2296
lpcox commentedon Apr 29, 2026
This is an operational credential issue — the
GOOGLE_API_KEYrepository secret needs to be rotated/replaced with a valid key.Code-side mitigations already in place in gh-aw-firewall:
Action needed: Rotate the
GOOGLE_API_KEYsecret in github/gh-aw repo settings.