[audit-workflows] Daily Audit 2026-05-20 — 5 failures + Codex/AI Moderator broken, OTLP shifts to 404

[github-actions[bot]]

Summary

Audit window: 2026-05-20 (24h) · Source: run §26191714235

• Runs: 78 total (67 success / 5 failure / 1 cancelled / 5 still in progress)
• Success rate: 85.9% (down from 88.9% on 05-19 and 92.8% on 05-17)
• Cost: $26.72 (1.09K turns, 744 action-minutes, 59.6M tokens, 336.9M effective)
• Engines: copilot 54 · claude 17 · codex 7
• Firewall: 1,967 allowed / 722 blocked (27% block rate)

Critical Issues

1. Codex / AI Moderator completely broken (high)

Run §26183420845 burned 16.8m with 0 productive turns. Two stacked errors:

Turn error: unexpected status 404 Not Found: Model not found gpt-5-codex-alpha-2025-11-07
Turn error: Missing environment variable: `OPENAI_API_KEY`

The model pin points to a non-existent variant and the fallback key isn't wired. Fix: update workflow frontmatter to a current Codex model and verify OPENAI_API_KEY plumbing.

2. Auto-Triage Issues 100% failure (high)

Observability insights flagged it as the day's failure hotspot — 2/2 runs failed at the agent step (§26184375104, §26181990934), each at 6 turns. Both runs also show the ERR_SYSTEM: rpc-messages.jsonl is present but zero bytes MCP telemetry capture failure.

3. OTLP exports now return 404 (was 401 on 05-17) (high)

Repeated across many runs: traces export: failed to send to ***: 404 Not Found (body: (empty)). The auth fix from 05-17 likely landed but the endpoint path is wrong — every Contribution Check / Auto-Triage agent step logged this. Verify Sentry+Grafana OTLP collector URLs.

4. Daily Code Metrics push_repo_memory rejected (medium)

Run §26184107476: agent succeeded ($4.20, 57 turns) but memory commit failed:

Patch additions size (64 KB, 65035 bytes) exceeds maximum allowed size (60 KB, 61440 bytes)

The agent's writes overflowed the 50 KB cap. Trim the per-run additions or raise max-patch-size.

5. PR review workflows blocking 40-59% of requests (medium)

Test Quality Sentinel, Matt Pocock Skills Reviewer, PR Code Quality Reviewer routinely block ~50%. OTLP Data Quality Validator: 59% blocked (highest, 20/34). Mostly api.githubcopilot.com:443 and (unknown) — either expand the allowlist or stop probing.

Trend Charts

Success rate has slid 7 points over three audits (92.8% → 88.9% → 85.9%). Failure mix is shifting from infra (telemetry, runtime) to workflow-specific (Auto-Triage broken, Codex misconfigured).

Today's $26.72 sits between the two prior audits; per-run cost is dominated by Claude workflows ($26.72 from 17 runs, ~$1.57/run avg). Three runs blew past $4 (Failure Investigator $5.58, Safe Output Tool Optimizer $4.93, Daily Code Metrics $4.20).

Per-workflow breakdown

Workflow	Runs	S/F	Engine	Tokens	Cost
Test Quality Sentinel	9	8/1	copilot	5.9M	–
Matt Pocock Skills Reviewer	9	8/1	copilot	6.5M	–
Design Decision Gate 🏗️	9	9/0	claude	4.1M	$5.72
PR Code Quality Reviewer	9	9/0	copilot	4.0M	–
Smoke CI	7	5/2	copilot	–	–
AI Moderator	6	4/2	codex	–	–
PR Sous Chef	4	4/0	copilot	1.4M	–
Auto-Triage Issues	2	0/2	copilot	0.4M	–
Daily Code Metrics	1	0/1	claude	4.9M	$4.20
[aw] Failure Investigator (6h)	1	1/0	claude	7.4M	$5.58
Daily Safe Output Tool Optimizer	1	1/0	claude	6.7M	$4.93
Contribution Check	1	0/1	copilot	1.7M	–

Top 10 firewall-pressure runs

All runs with >40% blocked + >10 total requests:

• Linter Miner (§26180896986): 79/174 blocked (45%)
• Daily Project Performance Summary Gen (§26190549711): 34/67 (51%)
• Test Quality Sentinel (§26191345725): 27/54 (50%)
• OTLP Data Quality Validator (§26186919756): 20/34 (59%)
• Daily Testify Uber Super Expert (§26182465277): 29/58 (50%)
• Test Quality Sentinel (§26187066401): 33/78 (42%)
• Daily Secrets Analysis Agent (§26181706924): 17/34 (50%)
• Chaos PR Bundle Fuzzer (§26190202701): 25/51 (49%)
• Matt Pocock Skills Reviewer (§26191345726): 14/29 (48%)
• Matt Pocock Skills Reviewer (§26191345877): 13/27 (48%)

DIFC-filtered tool calls (integrity)

10 filtered events this window: agent attempted list_issues / pull_request_read on resources tagged none:all or unapproved:all. Confirms DIFC is enforcing the configured threshold — not a bug. Examples: issues #33084, #33622, #33614, #33619; pr #33595.

Recommendations (priority order)

1. [high] Fix Codex AI Moderator — model pin gpt-5-codex-alpha-2025-11-07 returns 404; also missing OPENAI_API_KEY. Update model + verify secret. (fix-codex-openai-key-and-model)
2. [high] Fix Auto-Triage Issues — 100% failure rate (2/2); investigate agent-step exit ahead of any model downgrade. (fix-auto-triage-failure)
3. [high] Rotate OTLP endpoints (404) — 401 symptom from 05-17 is now 404; collector route appears stale. (rotate-otlp-endpoints-404)
4. [medium] Raise/trim push_repo_memory cap — Daily Code Metrics overflowed 60 KB by 4 KB; either trim agent writes or raise max-patch-size. (raise-push-repo-memory-patch-size)
5. [medium] Fix mcp-logs permissions — /tmp/gh-aw/mcp-logs/rpc-messages.jsonl: permission denied blocks MCP telemetry across multiple runs. (fix-mcp-rpc-jsonl-permissions)
6. [medium] Tune PR-review firewall allowlist — 40-59% block rates on review workflows + OTLP validator; expand or stop probing. (tune-pr-review-firewall-allowlist)
7. [medium] Stabilize Test Quality Sentinel prompt — turn count 0-41 (avg 17.7); execution-drift signal. (trim-prompt-test-quality-sentinel)

References

• Audit run §26191714235
• Codex/AI Moderator failure §26183420845
• Daily Code Metrics push failure §26184107476

Generated by 🔍 Agentic Workflow Audit Agent · ● 26M · ◷

• expires on May 21, 2026, 9:59 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #33873.