fix(litellm): cap-cooldown rotator + router cooldown + stop SpendLogs prompt bloat (#509, #510) - #527
Merged
Merged
Conversation
Three LiteLLM-side resilience changes (Helm/config only, all additive): - Storage: STORE_PROMPTS_IN_SPEND_LOGS=false. SpendLogs had grown to ~2.6GB from inline prompt/response text; cost/usage data is retained, only the raw payloads are dropped (#510 storage half). - Router cooldown (#510): add allowed_fails=2 + cooldown_time=60 to router_settings so a failing/429'd deployment is parked instead of re-picked immediately. num_retries/retry_after + fallback order unchanged. - Rotator cap-cooldown (#509): the codex-auth-rotator now tracks a per-account cooldown map in rotator_state.json. A usage_limit_reached 429 parks the active account for 3h (floor; ChatGPT Team caps are multi-hour-to-~3-day); an ambiguous 429 gets a short 5m park. Selection loop skips cooled accounts; if all are cooled it falls back to the least-recently-cooled one (never deadlocks). Old state files without 'cooldowns'/'last_label' still load. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019vuFLFHnmJKmMxosHSWA4P
samxu01
added a commit
that referenced
this pull request
Jun 29, 2026
…nded summarizer fan-out (#531) Anti-pattern guardrails for public exposure: per-key max_parallel_requests/rpm on issued virtual keys (G1), global request_timeout=120 for non-codex models (G4), bounded summarizer concurrency to stop the master-key fan-out flood (G2, re-fixes the PG-pool vector). Loop/failover guards (#508/#527/#509) untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019vuFLFHnmJKmMxosHSWA4P
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Code-half of the LLM-fleet resilience work (no new spend) + the storage fix.
#509 — rotator skips usage-capped accounts. The codex-auth-rotator now parks an account on a usage-cap 429 (
usage_limit_reached) for a 3h floor (envCODEX_CAP_COOLDOWN_SEC) and an ambiguous/transient 429 for 5m, persisting acooldownsmap in rotator_state. The selection loop skips parked accounts; if ALL are capped it falls back to the least-recently-cooled so codex always has some auth (never deadlocks). Backward-compatible with old state files (nocooldownskey → nothing skipped). The rate-limit callback already writes the exception text into the signal, so the cap-vs-transient classification works end-to-end.#510 — router deployment cooldown. Added
allowed_fails: 2+cooldown_time: 60to router_settings so a failing/429’d deployment is parked briefly instead of re-picked every request (complements the account-level cooldown). Fallback chain order unchanged.Storage.
STORE_PROMPTS_IN_SPEND_LOGS=false— SpendLogs had ballooned to ~2.6GB storing full prompt+completion text inline; cost/usage data is retained, only the raw text is dropped. (The dead SpendLogToolIndex was already truncated live; that population-disable patch stays.)The cross-cutting “agents paused — capacity” fleet UX is gateway-side and out of scope (noted as follow-up). Deploy carefully — the rotator is incident-prone; watch the litellm pod on rollout.
🤖 Generated with Claude Code