Pool keeps a held account broken (repeated websocket close 1011, healthy quota) until 'use this account' is clicked; reported as rate_limit_exceeded #5887

Summary

A pooled Codex account with healthy quota stopped answering. For ~66 minutes every request failed upstream with 502 … codex websocket closed before a Responses terminal event (close 1011). The pool then held the account and reported rate_limit_exceeded, saying the hold "clears on its own as the account recovers; no cooldown to lift and no account to switch". It did not clear on its own. Clicking "use this account" on the already-selected account fixed it immediately, with no credential change.

Environment

• opencodex 2.66.0 (service on Linux, hub mode). Clients: Codex CLI 0.157.0 on the host and on a remote machine through the hub.
• Pool: accountPoolStrategy: quota, autoSwitchThreshold: 98. Two Codex (pro) accounts:
  • A is selected and pinned (activeCodexAccountPinned), at ~17% weekly.
  • B is paused, at 100% weekly (mainAccountHardLock: blocked).
• ocx provider test openai → connected. ocx doctor → WHAM reachability ok.

Timeline (UTC, 2026-09-25)

• 22:37:08: first 502 upstream_server_error on account A (openai-<label> provider). The last 200 on A was at 22:38:33.
• 22:37–23:44: A gets 222 × 502, every one close 1011, cause=no-response-event, relayed=0, ~450–770 ms after send. This held across:
  • every conversation, including brand-new sessions;
  • request sizes from ~65 KB (a one-word prompt) to ~970 KB;
  • models gpt-6-luna, gpt-6-sol and gpt-5.6-sol.
• Meanwhile 12 × 429 on the openai provider:

  Codex account (account-…) is held after repeated upstream failures and the pool's recovery budget for this window is spent, so nothing was sent upstream. Retry after …Z. This clears on its own as the account recovers; no cooldown to lift and no account to switch.

• Probes after each "Retry after" time still failed with 1011.
• Quota reports for A stayed fresh and healthy throughout: weekly 17%, observed from response headers on the failing requests, so the requests did reach the backend. ocx account refresh openai showed no reauth need.
• 23:44:49: the owner clicked "use this account" on A in the dashboard (A was already selected). The next requests returned 200. The credential generation was unchanged (3 → 3).

Expected

When a held account's hold expires, the pool's recovery probe should do whatever "use this account" does: drop the account's cached upstream transport/session and conversation bindings, then probe fresh before routing traffic back. A pool pinned to one account with healthy quota should not stay broken until a human re-selects it.

Also

• The failure is reported as errorCode: rate_limit_exceeded / failureCause: rate-limit. An account-wide upstream failure (repeated 1011) with healthy quota is not a rate limit. Labelling it as one sent us looking for quota problems first. A distinct cause (e.g. upstream-account-failure) would help.
• No CLI command exposes the hold or recovery state for an account. ocx system diagnostics and ocx account list both show the account as selected and healthy.

Evidence available

usage.jsonl rows are available on request, with the account labels and conversation ids redacted.