Skip to content

[deep-report] PR Sous Chef: daily AIC guardrail cascading failure from one run's missing usage-accounting file #61137

Closed

Description

@github-actions

Problem

The PR Sous Chef workflow has failed 100% of its scheduled runs since ~2026-09-15T10:43Z (7/7 in a spot-checked window). Root cause, confirmed by reading the activation job logs for run 34969377719:

[daily-workflow-aic] AIC inspection failed: {"status":"transient_error","error":"Missing accounting for executed agent component in run 34952385840 (attempt 1, job 104326484539, conclusion failure): agent/token_usage.jsonl is missing; agent_usage.jsonl is missing; agent_usage.json is missing"}
##[error]Daily workflow AI Credits are unknown: ...

A single earlier run (34952385840, itself already reported as a workflow failure in issue #61081/#61082) had its agent job fail before writing any usage-accounting file. The daily-AIC guardrail (actions/setup/js/check_daily_aic_workflow_guardrail.cjs, invoked by activation) walks back over the last 24h of runs to total AI Credit usage, and sumCoveredComponents in actions/setup/js/daily_aic_component_coverage.cjs throws when it can't prove coverage for a completed, non-skipped component job. That "unrecoverable" throw fails activation for every subsequent run that includes run 34952385840 in its 24h lookback window — not just the poisoned run itself.

The code already has a carve-out for exactly this scenario, but it's incomplete: daily_aic_component_coverage.cjs:152 treats a failed agent job as zero-cost ("failed_before_accounting") only when the primary accounting file exists but is empty (candidateStates[0].state === "empty"). When the file is fully missing (state === "missing", as observed here — likely because the job failed before the artifact-producing step ever ran), that carve-out doesn't match, and the strict-coverage check throws instead, cascading the one bad run's failure onto the whole fleet for up to 24h.

Why it matters

This is a self-perpetuating outage: one failed run with a missing usage file blocks every subsequent scheduled run of the same workflow from ever executing its agent job, until the poisoned run ages out of the 24h window. This is currently live and actively blocking PR Sous Chef.

Suggested fix

In sumCoveredComponents (actions/setup/js/daily_aic_component_coverage.cjs, near line 152), extend the existing name === "agent" && job.conclusion === "failure" zero-cost carve-out to also accept candidateStates[0].state === "missing" (not only "empty"), since a failed job that never reached the accounting-write step is equally strong proof of no billable usage as one that wrote an empty file.

Data source

Live fleet-log spot-check (Step 2) + direct log read of run 34969377719's activation job (job 104381785440) + source read of actions/setup/js/daily_aic_component_coverage.cjs and check_daily_aic_workflow_guardrail.cjs, this cycle. Dedup-checked: #61081/#61082 are auto-generated per-run failure reports for the original poisoned run and do not mention or diagnose the cascading guardrail bug; no other open issue found via local issues snapshot grep or search_issues.

Suggested Agent: general coding agent (Node/JS, gh-aw actions internals)
Estimated Effort: Small (one-line condition change + regression test for the missing-file case in check_daily_aic_workflow_guardrail.test.cjs)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 🔬 Deep Report · claude · agent · 279.5 AIC · ⌖ 12.7 AIC · ⊞ 12.8K · ◷

  • expires on Sep 17, 2026, 4:50 AM UTC-08:00

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions