Skip to content

[aw-failures] [aw] Failure Investigator Report — 2026-08-09 (6h) #51545

Closed as not planned
4 of 4 issues completed
Closed as not planned
[aw-failures] [aw] Failure Investigator Report — 2026-08-09 (6h)#51545
4 of 4 issues completed
Labels
agentic-workflowsautomated-analysisautomationbugSomething isn't workingcookieIssue Monster Loves Cookies!

Description

@github-actions
github-actions
bot
opened on Aug 9, 2026

Fix the Step Name Alignment sandbox lockout first — it fails 100% of the time and never does any work

4 failed runs analyzed from the last 6 hours (2026-08-09 02:52–06:12 UTC). 3 distinct failure clusters + 1 expected security-gate rejection (not a bug). 0 of today's clusters are covered by an existing open issue. No open "Failure Investigator Report" parent existed (most recent, #50968, closed 2026-08-07) — new parent filed here per the P0-with-no-coverage rule. 1 P0 sub-issue filed below; 2 P1 gaps folded into this report (create_issue budget: 2/run).

Cluster summary

Sev Cluster Runs Status
P0 Step Name Alignment — sandbox denies every op (wc/jq/cp/Read) against its own 6.7MB input file, agent gives up after 32 turns §31294247934 Untracked — sub-issue filed below
P1 Semantic Function Refactoring — turns blow up 9→79, 3.71M tokens, exits 1 after a transient gh api network error triggers full-repo fallback exploration §31291416435 Untracked — folded into fix roadmap (budget)
P1 Daily Regression Audit — Kiro — hard ai_credits_rate_limit_error block, 0 turns executed, rejected at pre-activation §31291239673 Untracked — folded into fix roadmap (budget)
P2 Daily Container Image Security Scan — gate correctly failed the build on a real critical CVE / license violation §31297209717 Not a bug — informational only

Evidence

Step Name Alignment (P0) — audit cohort-matched this against baseline [§31239045503] (success, 56 turns). The failing run instead did fewer turns (32) before quitting — reason_codes: ["turns_decrease", "run_unsuccessful"]. The log shows the agent trying every angle to read /tmp/gh-aw/agent/step-alignment-input.json (6.7MB) and getting blocked each time:

wc in '/tmp/gh-aw/agent/step-alignment-input.json' was blocked... allowed working directories for this session: '/home/runner/work/gh-aw/gh-aw'
jq in '/tmp/gh-aw/agent/step-alignment-input.json' was blocked...
cp in '/tmp/gh-aw/agent/step-alignment-input.json' was blocked...

Read also refused the file outright at first (6.7MB > 256KB cap), and only Grep/paginated Read worked around it — but the agent burned its turn budget on the blocked wc/jq/cp attempts first every time. This is 100% reproducible: the workflow's own generated input lives in the one directory (/tmp/gh-aw/agent/) that general-purpose CLI tools aren't allowed to touch.

Semantic Function Refactoring (P1) — audit cohort-matched against baseline [§31236037799] (success, 9 turns). Failing run: 79 turns, 3.71M tokens, 45 requests, error_count: 1, ##[error]Process completed with exit code 1. audit-diff shows 19 new tool types appear only in the failing run (serena get_symbols_overview ×10, rg, gh api search/issues, gh api --paginate, python3 .aw_scan.py, ...) — none present in the baseline run. The log opens with gh api "search/issues?..." failing (error connecting to api.github.com), and the agent falls back to broad Serena/ripgrep exploration across the whole repo instead of retrying the narrow lookup. This is the same shape as the already-fixed-once #49886 ("Raise Semantic Function Refactoring turn budget") and the just-merged 3a2315f ("Raise Avenger turn budget for self-repair runs", #51491) — the turn-budget fix landed for Avenger but wasn't mirrored here, or has regressed.

Daily Regression Audit — Kiro (P1) — audit shows turns=0, tool_types=0, duration=3.0m, no baseline match available. The job-outputs list at the end of the run includes Set output 'ai_credits_rate_limit_error' — the run was rejected at the pre-activation AI-credits gate before the agent executed a single turn.

Daily Container Image Security Scan (P2) — the "Enforce critical vulnerability and license gates" step ran exactly as designed: grep -qE ': error: \[Critical\]' "$output" matched, so it printed ##[error]Critical vulnerabilities detected in container images. and exited 1. This is the gate catching a real finding, not an automation defect.

Existing issue correlation

Fix roadmap

P0 — do this first: Get Step Name Alignment off the sandbox blocklist. Either widen the session's allowed working directories to include /tmp/gh-aw/agent/ for this workflow, or have a pre-agent step stage step-alignment-input.json inside the repo checkout before the CLI runs. Verify by confirming the next 3 scheduled runs have zero permission_denied entries.

P1 — raise the Semantic Function Refactoring turn budget. Apply the same fix #51491 gave Avenger. Also wrap the opening gh api search/issues call in a retry/backoff so one transient network blip doesn't cascade into full-repo Serena/ripgrep exploration. Verify with audit on the next run: turns should land back near the ~9-turn baseline.

P1 — fix the Kiro AI-credits scheduling collision. Check whether Daily Regression Audit — Kiro's 02:52 UTC slot is starved by a concurrent AIC-heavy job; stagger the schedule or give Kiro its own credit allocation so pre-activation stops rejecting it outright.

P2 — no automation fix needed. Route the container scan's critical CVE / license violation to whoever owns that base image.

Sub-issues created

  • P0 sub-issue for the Step Name Alignment sandbox lockout — filed below, linked to this parent.

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 119.7 AIC · ⌖ 47 AIC · ⊞ 5.5K · ◷

  • expires on Aug 15, 2026, 11:19 PM UTC-08:00

Fix the Step Name Alignment sandbox lockout first — it fails 100% of the time and never does any work

4 failed runs analyzed from the last 6 hours (2026-08-09 02:52–06:12 UTC). 3 distinct failure clusters + 1 expected security-gate rejection (not a bug). 0 of today's clusters are covered by an existing open issue. No open "Failure Investigator Report" parent existed (most recent, #50968, closed 2026-08-07) — new parent filed here per the P0-with-no-coverage rule. 1 P0 sub-issue filed below; 2 P1 gaps folded into this report (create_issue budget: 2/run).

Cluster summary

Sev Cluster Runs Status
P0 Step Name Alignment — sandbox denies every op (wc/jq/cp/Read) against its own 6.7MB input file, agent gives up after 32 turns §31294247934 Untracked — sub-issue filed below
P1 Semantic Function Refactoring — turns blow up 9→79, 3.71M tokens, exits 1 after a transient gh api network error triggers full-repo fallback exploration §31291416435 Untracked — folded into fix roadmap (budget)
P1 Daily Regression Audit — Kiro — hard ai_credits_rate_limit_error block, 0 turns executed, rejected at pre-activation §31291239673 Untracked — folded into fix roadmap (budget)
P2 Daily Container Image Security Scan — gate correctly failed the build on a real critical CVE / license violation §31297209717 Not a bug — informational only

Evidence

Step Name Alignment (P0) — audit cohort-matched this against baseline [§31239045503] (success, 56 turns). The failing run instead did fewer turns (32) before quitting — reason_codes: ["turns_decrease", "run_unsuccessful"]. The log shows the agent trying every angle to read /tmp/gh-aw/agent/step-alignment-input.json (6.7MB) and getting blocked each time:

wc in '/tmp/gh-aw/agent/step-alignment-input.json' was blocked... allowed working directories for this session: '/home/runner/work/gh-aw/gh-aw'
jq in '/tmp/gh-aw/agent/step-alignment-input.json' was blocked...
cp in '/tmp/gh-aw/agent/step-alignment-input.json' was blocked...

Read also refused the file outright at first (6.7MB > 256KB cap), and only Grep/paginated Read worked around it — but the agent burned its turn budget on the blocked wc/jq/cp attempts first every time. This is 100% reproducible: the workflow's own generated input lives in the one directory (/tmp/gh-aw/agent/) that general-purpose CLI tools aren't allowed to touch.

Semantic Function Refactoring (P1) — audit cohort-matched against baseline [§31236037799] (success, 9 turns). Failing run: 79 turns, 3.71M tokens, 45 requests, error_count: 1, ##[error]Process completed with exit code 1. audit-diff shows 19 new tool types appear only in the failing run (serena get_symbols_overview ×10, rg, gh api search/issues, gh api --paginate, python3 .aw_scan.py, ...) — none present in the baseline run. The log opens with gh api "search/issues?..." failing (error connecting to api.github.com), and the agent falls back to broad Serena/ripgrep exploration across the whole repo instead of retrying the narrow lookup. This is the same shape as the already-fixed-once #49886 ("Raise Semantic Function Refactoring turn budget") and the just-merged 3a2315f ("Raise Avenger turn budget for self-repair runs", #51491) — the turn-budget fix landed for Avenger but wasn't mirrored here, or has regressed.

Daily Regression Audit — Kiro (P1) — audit shows turns=0, tool_types=0, duration=3.0m, no baseline match available. The job-outputs list at the end of the run includes Set output 'ai_credits_rate_limit_error' — the run was rejected at the pre-activation AI-credits gate before the agent executed a single turn.

Daily Container Image Security Scan (P2) — the "Enforce critical vulnerability and license gates" step ran exactly as designed: grep -qE ': error: \[Critical\]' "$output" matched, so it printed ##[error]Critical vulnerabilities detected in container images. and exited 1. This is the gate catching a real finding, not an automation defect.

Existing issue correlation

Fix roadmap

P0 — do this first: Get Step Name Alignment off the sandbox blocklist. Either widen the session's allowed working directories to include /tmp/gh-aw/agent/ for this workflow, or have a pre-agent step stage step-alignment-input.json inside the repo checkout before the CLI runs. Verify by confirming the next 3 scheduled runs have zero permission_denied entries.

P1 — raise the Semantic Function Refactoring turn budget. Apply the same fix #51491 gave Avenger. Also wrap the opening gh api search/issues call in a retry/backoff so one transient network blip doesn't cascade into full-repo Serena/ripgrep exploration. Verify with audit on the next run: turns should land back near the ~9-turn baseline.

P1 — fix the Kiro AI-credits scheduling collision. Check whether Daily Regression Audit — Kiro's 02:52 UTC slot is starved by a concurrent AIC-heavy job; stagger the schedule or give Kiro its own credit allocation so pre-activation stops rejecting it outright.

P2 — no automation fix needed. Route the container scan's critical CVE / license violation to whoever owns that base image.

Sub-issues created

  • P0 sub-issue for the Step Name Alignment sandbox lockout — filed below, linked to this parent.

Recurrence + new-cluster update — 2026-08-09 (13:10–17:39 UTC pass)

Fix the two new untracked P1s below now — PR Sous Chef's safe_outputs job and Contribution Check's AWF-proxy block are both new gaps this pass, and #51033's "No model available" bug just spread to a 4th workflow. 5 failed runs analyzed, 4 distinct failure clusters: 1 already tracked (updated in place), 2 new (sub-issues filed below), 1 isolated P2 (watching, not filing yet).

Cluster summary

Sev Cluster Runs Status
P1 PR Sous Chef — resolve_pull_request_review_thread fails on stale/already-resolved GraphQL node IDs, fails the safe_outputs job despite agent success §31322214642, §31319713324 Untracked — sub-issue filed (#aw_prsc)
P1 Contribution Check — AWF API-proxy denies git fetch + safeoutputs create_issue, harness treats it as non-retryable and gives up on attempt 1/4 §31315109276 Untracked — sub-issue filed (#aw_ccheck)
P1 Linter Miner — Copilot Task-tool No model available, identical signature to #51033, now a 4th confirmed affected workflow §31326676901 Tracked — #51033 updated with recurrence evidence
P2 Design Decision Gate — push_to_pull_request_branch fails to pin a PR branch that no longer resolves locally (fatal: Needed a single revision), after the PR checkout itself already warned/failed; single, isolated occurrence §31320740338 Untracked, isolated — no sub-issue filed (below threshold)

Evidence

PR Sous Chef (P1) — both runs show create_issue / update_pull_request / add_comment messages succeeding in the same batch while every resolve_pull_request_review_thread call fails with GraphQL Could not resolve to a node with the global id of 'PRRT_...' or plain Not Found. Same shape both times: the thread ID is stale by the time safe_outputs replays it, most likely because the thread was already resolved before the safe-outputs job ran.

Contribution Check (P1) — failureClass=permission_denied, 19 denials, harness log: AWF API proxy is blocking requests — not retrying (non-retryable guard condition). Denied commands were ordinary and workflow-critical (git fetch origin pull/51582/head:pr-51582, safeoutputs create_issue), not sandbox-escape attempts — all 3 PR review comments still posted successfully; only the run's own summary issue got dropped, and the harness never used its remaining 3 retries.

Linter Miner (P1, folded into #51033) — identical signature to the existing tracked issue: Error: No model available. Check policy enablement under GitHub Settings > Copilot on all 4 harness attempts for a general-purpose Task subagent. #51033 updated with this as a newly confirmed 4th affected workflow (beyond PR Code Quality Reviewer and Daily Agent of the Day Blog Writer).

Design Decision Gate (P2) — push_to_pull_request_branch errored Failed to pin branch 'purelock/sameexpr-addallowed-rpcentry-tests-5ef3ea127862592b' before bundle generation: ERR_SYSTEM: fatal: Needed a single revision, preceded by ##[warning]Checkout failed (expected for closed PR) and fatal: couldn't find remote ref. Single occurrence — the evaluated PR's branch was gone by the time the safe-output tool tried to push to it. Watching for recurrence before filing a fix.

Existing issue correlation

Fix roadmap

P1 — stop resolve_pull_request_review_thread from failing the whole safe_outputs job. Skip (don't fail) when the GraphQL node is already gone — see #aw_prsc.

P1 — fix the Contribution Check AWF-proxy false block. Verify the sandbox allowlist covers git fetch and safeoutputs create_issue for this workflow, and make the guard retryable instead of an instant give-up — see #aw_ccheck.

P1 — escalate #51033 again. 4th affected workflow in 3 days; apply the non-fatal-skip stopgap now instead of waiting on the upstream Copilot policy fix.

P2 — watch, don't fix yet. Design Decision Gate's branch-pin failure is a single occurrence tied to a PR whose branch disappeared mid-run; revisit if it recurs.

Sub-issues created (this pass)

  • #aw_prsc — PR Sous Chef resolve_pull_request_review_thread failures (P1)
  • #aw_ccheck — Contribution Check AWF-proxy permission_denied block (P1)

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 119.7 AIC · ⌖ 47 AIC · ⊞ 5.5K · ◷

Fix the Copilot harness segfault first — it's the only P0 this pass

4 failed runs analyzed from the last 6 hours (2026-08-10 06:55–13:11 UTC). 3 distinct failure clusters: 1 already tracked (left untouched, still recurring), 2 new (sub-issues filed below), 1 unconfirmed (log access blocked — watching, not filing).

Cluster summary

Sev Cluster Runs Status
P0 Agent Performance Analyzer - Meta-Orchestrator — Copilot CLI harness segfaults (exit 139) right after the pre-flight /reflect fetch, before Copilot even spawns; first failure in 11 daily runs §31390296030 Untracked — sub-issue filed (#aw_seg1)
P1 Claude Code User Documentation Review — Claude harness returns API Error: 400 ... not valid JSON on the turn immediately after a permission_denied rejection, burning the last retry; first failure in 8 daily runs §31388621726 Untracked — sub-issue filed (#aw_json4)
P2 Daily Max Ai Credits Test — same authentication_failed HTTP 403 billing block as #51589 §31383727949 Tracked — #51589, still unresolved, left untouched
P1 Instructions Janitor — job failed but the last-50-line log tail is 100% post-job cleanup with no error visible; first failure in 8 daily runs §31372258689 Unconfirmed — log fetch blocked this pass, no sub-issue filed (see below)

Evidence

Agent Performance Analyzer (P0) — crash trace: [copilot-harness] awf-reflect: fetching (apiproxy/redacted) → /tmp/awf-cmd-1.sh: line 29: 80 Segmentation fault→ exit 139.audit-diff` against the nearest successful run (§31314425506) shows zero firewall/network drift (0 new domains, 0 anomalies) — rules out a sandbox/policy change; this is a raw process crash in the node-based harness, not network-triggered. Full root cause and fix plan in #aw_seg1.

Claude Code User Documentation Review (P1) — a compound Bash command gets rejected (permission_denied, decision_reason_type":"subcommandResults"), and the very next turn (attempt 4/4, the last retry) the Anthropic API rejects the request body: API Error: 400 The request body is not valid JSON. The harness's own classifier logs permissionDeniedCount=0 — it doesn't even recognize what just happened, so this 400 burns the run's final retry with no fallback. Full root cause and fix plan in #aw_json4.

Daily Max Ai Credits Test (P2, tracked) — matches #51589's exact signature (Copilot-requests proxy authentication_failed HTTP 403). No new information this pass; issue already documents 5+ consecutive prior days and remains open. Not closing — still broken.

Instructions Janitor (P1, unconfirmed) — the pre-fetched log tail is only post-job git/cleanup steps; the actual failure happened earlier in the run and isn't in the truncated window. Both follow-up attempts to get more signal failed this pass: gh run view --log-failed returned error connecting to api.github.com (log-blob fetch blocked from this sandbox) and audit timed out after 60s (context deadline exceeded). Filed missing_data for this gap rather than guessing a root cause. First failure in 8 daily runs — recommend a maintainer with full log access pull §31372258689 directly; will re-open investigation and file a sub-issue if it recurs or if full logs surface a concrete signature.

Existing issue correlation

Fix roadmap

P0 — fix the Copilot harness segfault now. Capture a core dump around the /reflect fetch to find the crash site, then make a harness-process SIGSEGV retryable instead of a hard job failure. See #aw_seg1.

P1 — fix the Claude harness's post-denial 400. Stop the CLI from serializing a broken request body after a permission_denied tool result, and teach the retry classifier to recognize this 400 as retryable. See #aw_json4.

P1 — re-run Instructions Janitor with full log access. This pass's sandbox couldn't reach the log blob store or complete an audit call in time; a maintainer needs to pull §31372258689 directly to get the real error before this can be triaged further.

P2 — no new action. #51589 remains the correct tracker for Daily Max Ai Credits Test; keep it open until Copilot billing is fixed.

Sub-issues created (this pass)

  • #aw_seg1 — Agent Performance Analyzer Copilot CLI harness segfault (P0)
  • #aw_json4 — Claude Code User Documentation Review invalid-JSON 400 after permission_denied (P1)

References:

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agentic-workflowsautomated-analysisautomationbugSomething isn't workingcookieIssue Monster Loves Cookies!

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions