Skip to content

[aw-failures] [P1] Fix Copilot billing 403 masking Daily Max Ai Credits Test — 5+ consecutive daily failures #51589

Description

@github-actions

Problem

Fix the Copilot-requests proxy auth check before it burns another day of test signal — every scheduled run of Daily Max Ai Credits Test has failed on an authentication_failed HTTP 403 from the gh-aw Copilot API proxy for at least the last 5 consecutive days, never once reaching the max-ai-credits: 1 guardrail logic the workflow exists to validate.

Affected workflows and runs

  • Daily Max Ai Credits Test (.github/workflows/daily-max-ai-credits-test.lock.yml) — 100% failure, same signature, on every run checked:

Probable root cause

Every attempt dies the same way, before any credit-limit logic can engage:

Authentication failed with provider at (172.30.0.30/redacted) (HTTP 403).
  Check your COPILOT_PROVIDER_API_KEY or COPILOT_PROVIDER_BEARER_TOKEN.
[copilot-harness] attempt N failed: exitCode=1 failureClass=authentication_failed isAuthenticationFailedError=true ...
[copilot-harness] ...: Copilot requests authentication failed through the gh-aw API proxy (HTTP 403, model=claude-sonnet-5). This workflow is using permissions.copilot-requests: write, so Copilot requests must be allowed through your organization's centralized Copilot billing configuration. Verify that copilot-requests: write is granted to the workflow or job and that Copilot org billing is enabled for your organization.

This is not the workflow's intended ai_credits_rate_limit_error / max_ai_credits_exceeded outcome (which is explicitly excluded from report-failure-as-issue because it's the expected "success" case for this test). It is a distinct authentication_failed billing/policy rejection, so this bug has been silently invalidating the test's actual purpose since at least 2026-08-05.

audit-diff between today's run and the prior day's run (§31254595629 vs §31310376315) shows zero drift in firewall/network behavior — same signature, same failure point, confirming this is a stable, ongoing condition rather than a one-off blip.

Not a duplicate of #51033 (that signature is No model available from Task-tool subagent delegation — a different failureClass entirely) or #51547 (sandbox file-access lockout — unrelated).

Proposed remediation

  1. Verify Copilot org billing / copilot-requests: write centralized policy grants cover this workflow's job — this looks like an entitlement gap specific to this workflow's permission or token configuration, not a code bug in gh-aw itself.
  2. If the entitlement is confirmed correct, escalate to the Copilot API proxy team with the reproducing run IDs above — the 403 fires on the very first request of every run, 5 days running.
  3. Regardless of root cause, make the harness classifier retry authentication_failed distinctly from partial_execution — right now it burns a 5s sleep + full second attempt before giving up, wasting ~2x the run's actual work time on a condition it already knows won't recover mid-run.

Success criteria

  • Next 3 scheduled runs of Daily Max Ai Credits Test reach the max-ai-credits cutoff logic (i.e., fail via isCAPIQuotaExceededError/isInvocationCapExceeded/credit-limit signature, not authentication_failed).
  • Zero authentication_failed / HTTP 403 entries from this workflow across the next 3 days.

Related to #51545
Related to #51545

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 101.8 AIC · ⌖ 47.5 AIC · ⊞ 5.3K ·

  • expires on Aug 16, 2026, 5:18 AM UTC-08:00

Update — 2026-08-15 11:00-13:00 UTC investigation (6h)

Still not fixed — escalate now. This is the 7th consecutive day of the identical authentication_failed HTTP 403 killing every run before the max-ai-credits guardrail logic ever engages.

New occurrence: §31881546024 — 2026-08-15T11:14Z, same signature confirmed verbatim in agent-stdio.log:

Authentication failed with provider at (172.30.0.30/redacted) (HTTP 403).
[copilot-harness] attempt 2: Copilot requests authentication failed through the gh-aw API proxy (HTTP 403, model=claude-sonnet-5, stage=starting the Copilot CLI request). — not retrying

No drift from the original report — same proxy, same billing/policy rejection, same point of failure on both harness attempts. Success criteria from the original report (3 clean runs reaching the credit-limit logic) are still unmet after 7 days; treat this as beyond "monitor" and escalate directly to the Copilot API proxy/billing owner.> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 222 AIC · ⌖ 6.48 AIC · ⊞ 5.6K ·


New recurrence — 2026-08-16

Still happening. Run §31943793516 (2026-08-16 11:14 UTC) failed the same way: agent job fails at Execute GitHub Copilot CLI. Tail logs only capture MCP backend/tools-registration startup noise, not the actual billing/auth error — same log-capture gap as before. No fix has landed yet; keeping this open.> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 99.9 AIC · ⌖ 7.98 AIC · ⊞ 5.6K ·

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions