fix(cli): a quota stall was classified as a runtime blip and probed every 5s

[lilyshen0722]

Found by @ux-lead while diagnosing the 2026-08-18 fleet stall (in-pod, alongside #993). Their diagnosis is exact; this is the fix plus a control-verified regression test.

The defect

QUOTA_RE (cli/src/lib/spawn-retry.js:26) matched usage limit but not session limit. The Claude CLI's exhaustion wording is:

claude exited with code 1: You've hit your session limit · resets 4am (America/Los_Angeles)

That misses every alternative in the pattern, so classifySpawnFailure fell through to RUNTIME — the weakest class, with the shortest backoff. Observed live on my own seat: 5.6s → 11.2s → 1.1m, against a provider that would not answer until 4am. QUOTA opens the circuit on the first failure at the full SPAWN_RETRY_MAX_MS ceiling, which is the right response to a quota that resets at a wall-clock hour.

Why it's worth more than a one-word diff

usage limit was already in the list — it is Claude's other exhaustion wording. So the fleet stalled for an hour on a string one word away from a pattern we already had.

That makes this the second instance of the same shape. The comment directly above QUOTA_RE documents the first:

out of credits is codex's exact wording for an exhausted workspace balance … Without it that outage classified as RUNTIME and drew the shortest backoff — observed live on 2026-08-03 before this pattern was added.

The pattern list is a per-provider allowlist that only ever grows after an outage has already been misclassified. The comment now records both dates and says that plainly, so the next person adding to it knows they are the third.

What this deliberately does not do

Not loosened to a bare limit. QUOTA is tested before RATE_LIMIT, so a pattern that broad would swallow every rate-limit error into a 15-minute cooldown — turning a fix for one misclassification into a worse one in the other direction. The test asserts that directly rather than leaving it to review: 429 rate limit exceeded and Too many requests must still classify RATE_LIMIT after the widening.

Proof

Run red first against the live string: both defect tests failed (classifying RUNTIME), and the rate-limit guard passed before and after — it is a control, not a beneficiary. Full cli suite green: 21 suites, 298 passed.

Scope — this does not rescue the lost events

Stated because the two findings arrived together and it would be easy to read this as the fix for both. The events lost in the 2026-08-18 stall die in agentEventService.garbageCollect() (#993), where the requeue and the stale-pending delete run in the same pass. That happens regardless of which backoff the seat chose. Correcting the class reduces wasted spawn attempts during an outage; it does not change what happens to the queue.

🤖 Generated with Claude Code

[lilyshen0722]

Sharpening this PR's own scope note, from @sprint-review's finding in-pod. I wrote that correcting the class "reduces wasted spawn attempts during an outage." That's true but weak. The stronger and more useful statement:

The retry ladder has no effect whatsoever on the failed event's fate — at any delay.

• AgentEventService.list() hard-filters status: 'pending' (agentEventService.ts:1094).
• A failed spawn leaves the event delivered — the wrapper's only event-state call is the ack at agent.js:1109. There is no nack and no release. (claimKeeper.release() at :881 is the ADR-018 message claim, a different mechanism on a different object.)
• So from the moment a spawn fails, that event is invisible to every subsequent poll by construction. The seat cannot re-fetch it whether it probes in 5.1s or in 15 minutes.
• Only garbageCollect()'s requeue pass can move it back to pending — deliveredAt < now-10min on a */10 cron, which its own comment describes as uniform over 10–20 min, mean ~15.
• And if it is older than the 30-minute stale threshold when that requeue fires, the delete in the same pass destroys it (garbageCollect() requeues events to 'pending' and deletes them in the same pass — deterministic for any delivery lag ≥20min #993).

So the backoff governs only how quickly the seat picks up new work. That is still worth fixing — a 5-second probe loop against a provider that will not answer until 4am is pure waste, and it is the difference between one spawn attempt and hundreds across a fleet — but it is not, and cannot be, a retry of the thing that failed.

Related AX defect, which I'll file separately rather than fold in here. The log line this PR governs reads:

… (1 consecutive) — event 6a842eb896408f264d9a4846 remains unacked; retry scheduled, next probe in 5.1s

"retry scheduled" + the event's own ID reads as we will try this event again in 5.1s. We will not. We will poll, that event will not be in the response, and it returns only when the cron requeues it. The line names the one event it is specifically not retrying — exactly the shape docs/development/agent-experience-audit.md exists to record.

[lilyshen0722]

Post-merge finding by @sprint-review: this PR has a cost on the fetched-then-failed path that I did not see, and it is the opposite of the "rescues nothing" note I shipped it with.

I wrote that correcting the quota class "does not change what happens to the queue." That is true of the events already in flight and wrong about the direction. The mechanism is delivery lag, not probe count.

Let L = time from enqueue to first fetch. The GC requeues a stuck delivered row at deliveredAt + 10..20 (10-minute gate, */10 cron) and deletes pending rows older than 30 minutes in the same pass. So the row is destroyed at its own requeue iff:

L + 10..20 > 30      ⇔      L > 10..20

• RUNTIME ladder (5.6s, 11.2s, 1.1m, …): L ≈ 0 early in a stall. Requeue lands at age 10–20 — under the line. The row survives that pass and is redelivered.
• QUOTA ceiling (900s from failure Add basic unit tests for backend functions #1, this PR): L up to 18 min. Requeue lands at age 28–38 — over the line on most cron landings. The row is destroyed by the pass that was supposed to rescue it.

One narrowing, which makes the cost specific rather than smaller. For outages longer than ~30 minutes the row dies either way; fast polling only buys it one extra redelivery before the following pass takes it. What this PR actually costs is recoverability in short stalls — provider back at T+15, where the RUNTIME ladder had the event requeued and retryable and this one has not fetched it yet.

Not proposing a revert. Probing a dead provider every 5 seconds is its own harm, the classification is correct, and the 180× misclassification it fixes is real. But this makes #995 the second change that depends on #1001 rather than being independent of it, and both dependencies point the same way: the 30-minute pending delete is what turns every latency decision into a data-loss decision.

Recorded here rather than only in the pod, because the "rescues nothing" line is in this PR's merged body and a reader would otherwise inherit it.