[aw-failures] Multi-Device Docs Tester exhausts 100-LLM-invocation API cap during multi-device sweep #31651

Problem statement

The scheduled run of Multi-Device Docs Tester failed today with an Anthropic API-level rate-limit error after the harness's last retry: API Error: Request rejected (429) · Maximum LLM invocations exceeded (100 / 100). This is not the workflow-level --max-turns 80 cap (the agent only completed 13 turns) nor a wall-clock timeout (the step's timeout-minutes: 30 was not reached). It is an upstream account/key invocation quota the harness cannot retry past.

The workflow is structurally heavy: it spins up the docs Astro dev server, then drives Playwright through five viewports (iPhone 12, Galaxy S21, iPad, FHD desktop, HD desktop) against multiple pages (/, /quick-start/, ...). In this run the agent issued 92 Bash tool calls and many Playwright MCP calls before the API ran out of invocations mid-loop.

Affected workflow and run

Workflow source: .github/workflows/daily-multi-device-docs-tester.md (compiled to .lock.yml). Configured limits: max-turns: 80, timeout-minutes: 30.

Evidence

{"type":"assistant","message":{"model":"<synthetic>","content":[{"type":"text","text":"API Error: Request rejected (429) · Maximum LLM invocations exceeded (100 / 100)."}]},"error":"rate_limit"}
{"type":"result","is_error":true,"api_error_status":429,"num_turns":1,"terminal_reason":"completed"}
[claude-harness] attempt 4 failed: isRateLimitError=true isMaxTurnsExit=false hasOutput=true retriesRemaining=0
[claude-harness] all 3 retries exhausted — giving up (exitCode=1)
[claude-harness] awf-reflect: models fetch returned 429 for (apiproxy/redacted)

Install docs dependencies
Start docs dev server in background           (×7 retries: server-readiness loop)
Check playwright-cli help / commands
Resize to iPhone 12  → navigate → snapshot
Galaxy S21          → navigate → snapshot
iPad                → navigate → snapshot
FHD desktop         → navigate → snapshot
HD desktop          → navigate → snapshot
Inspect right-sidebar / computed styles
Close playwright and dev server
(then a second full pass appears to begin again — Install docs dependencies, Start dev server, ...)

Probable root cause

1. Dev-server readiness is unreliable. The agent makes seven separate "Start docs dev server / Wait and retry / Check server status" calls before the page is reachable. Each retry costs an LLM turn (and several inference requests once the harness retries).
2. Playwright workflow is fully agentic. Each device × page combination is driven by the model rather than a deterministic post-step script, so the per-device cost compounds.
3. Harness retries on rate-limit re-enter the same expensive loop, burning invocations against a still-throttled key.

Proposed remediation

Pick one or combine:

• Move dev-server startup to a pre-agent deterministic step that polls until (localhost/redacted) returns 200, so the agent doesn't burn inference on readiness loops. (runtimes.stepsorsteps:` block before the agent invocation.)
• Replace the per-viewport sweep with a deterministic Playwright script (playwright test --project=mobile,desktop,...) and have the agent only summarize results.
• Detect 429 / "Maximum LLM invocations exceeded" in the harness's retry classifier and treat it as non-retriable (currently it retries 3 more times, multiplying the invocations charged before final exit).
• Short-term mitigation: lower max-turns from 80 to ~30 to fail fast and reduce upstream invocation usage when the workflow inevitably loops.

Success criteria

• A clean schedule run of Multi-Device Docs Tester completes within 30 minutes without 429.
• The agent issues ≤ ~40 tool calls (down from 92) thanks to deterministic readiness + Playwright script.
• The harness no longer issues 3 retries when the API returns Maximum LLM invocations exceeded.

Confidence and unknowns

High confidence that the immediate cause is the upstream LLM-invocations cap and that deterministic readiness + scripted Playwright would slash the inference cost. Unknown whether the 100/100 cap is a per-workflow-run quota imposed by this repo's Anthropic key configuration, a per-hour account quota, or a daily one — worth confirming before deciding whether to raise the cap or restructure the workflow.

References

• Parent failure report: [aw-failures] [aw] Failure Report 2026-05-11 (6h): aw-portfolio-yield workflow defect blocks compile + own runtime + recurrences [Content truncated due to length] #31455
• Run: §25717436849
• Workflow source: .github/workflows/daily-multi-device-docs-tester.md
  Related to [aw-failures] [aw] Failure Report 2026-05-11 (6h): aw-portfolio-yield workflow defect blocks compile + own runtime + recurrences [Content truncated due to length] #31455

Generated by [aw] Failure Investigator (6h) · ● 15.3M · ◷

• expires on May 19, 2026, 7:51 AM UTC

6h investigator update — 2026-05-13 13:31 UTC

Daily Rendering Scripts Verifier hit the same Maximum LLM invocations exceeded (100 / 100) cap within the 6h window. Adding as another affected workflow alongside Multi-Device Docs Tester.

• Run §25787700817 (2026-05-13 08:30 UTC), 15.9m wall, 93 turns, agent stop reason stop_sequence, final result event:
  • api_error_status: 429
  • result: "API Error: Request rejected (429) · Maximum LLM invocations exceeded (100 / 100)."
  • total_cost_usd: $5.90, cache_read_input_tokens: 8.6M
  • Engine: Claude (claude-opus-4-7[1m])
• Two consecutive daily failures: 25722812301 (2026-05-12) and 25787700817 (2026-05-13); prior daily runs (2026-05-07 through 2026-05-11) were successes — so this is a fresh recurrence on a workflow that had been stable.
• Side-channel evidence consistent with [aw-failures] Claude workflows hit max-turns burning every turn on Bash permission denials (Step Name Alignment, Design Decision Gate) #31178: the run's final result event lists many permission_denials on Bash tool calls (jq/cat/cd/find/wc/grep over /tmp/gh-aw/logs-cache/ and the parser harness), so wasted-turn pressure from denied tools is contributing alongside the absolute LLM cap.

Suggests the LLM-invocation cap mitigation should cover Verifier-style workflows that crawl many runs, not only Multi-Device Docs Tester.

Generated by [aw] Failure Investigator (6h) · ● 28.7M · ◷