diff --git a/docs-site/src/content/docs/fr/guides/combos.md b/docs-site/src/content/docs/fr/guides/combos.md index 8a9398988f6..ebd6023df2a 100644 --- a/docs-site/src/content/docs/fr/guides/combos.md +++ b/docs-site/src/content/docs/fr/guides/combos.md @@ -374,3 +374,7 @@ gérer un refus de politique ou corriger l’origine de la demande rejetée. Les Exception aux erreurs 400 terminales : un rejet structuré de `user`, une valeur non prise en charge pour `reasoning.effort`/`reasoning_effort`, un rejet d’entrée d’image propre à un modèle (`param: input`), ou le refus d’un média d’entrée que la route ne peut pas lire (`unsupported_input_modality`) peut faire passer le combo à la cible admissible suivante avant le début de la sortie, sans délai de refroidissement. Le refus de sécurité, l’annulation et une sortie déjà commencée restent non rejouables. [Canonical compatibility details](/guides/combos/#request-local-target-compatibility). + +## Réservation d’une tentative de récupération + +Une tentative de récupération activée par `cooldownReprobe: { enabled: true }` appartient uniquement à la requête qui l’a réservée ; les autres requêtes restent soumises aux délais de refroidissement enregistrés. La réservation est conservée pendant l’attente et la sélection JEV. Une cible redevenue utilisable pendant l’attente peut être réessayée si son délai enregistré n’a pas changé. Un nouvel enregistrement annule la réservation de cette cible. Un succès ne supprime que le délai inchangé de la cible ayant répondu ; les cibles non essayées et les nouveaux échecs conservent leurs délais. La désactivation du fournisseur, les quotas, l’admissibilité de la requête et l’annulation restent vérifiés. Voir les [réglages et limites](/guides/combos/#retry-all-cooled-targets-opt-in). diff --git a/docs-site/src/content/docs/guides/combos.md b/docs-site/src/content/docs/guides/combos.md index d32a8dcf122..7d69cc7b5e3 100644 --- a/docs-site/src/content/docs/guides/combos.md +++ b/docs-site/src/content/docs/guides/combos.md @@ -1068,3 +1068,10 @@ enforced. Explicit provider Retry-After values may therefore be retried early on when the operator enables this option. Preview/list operations never clear cooldowns. The interval is tracked per combo in memory and resets on proxy restart or combo selection-state reset. Existing cooldown-wait behavior is unchanged when disabled. + +A recovery pass belongs only to the request that reserved it; other requests still +observe the saved cooldowns. That reservation follows the request through cooldown +waits and JEV selection. A target that becomes usable again during a wait can still +be retried if its cooldown has not changed. A newer cooldown cancels that target's +reservation. Success clears only the unchanged cooldown of the target that answered; +untried targets and newer failures keep their cooldowns. diff --git a/docs-site/src/content/docs/ja/guides/combos.md b/docs-site/src/content/docs/ja/guides/combos.md index cb31f349e60..70c327e76d1 100644 --- a/docs-site/src/content/docs/ja/guides/combos.md +++ b/docs-site/src/content/docs/ja/guides/combos.md @@ -273,3 +273,7 @@ ocx combo remove --yes 通常の 400 エラーは終了扱いですが、`user` の明示的な拒否、`reasoning.effort`/`reasoning_effort` の非対応値、モデル固有の画像入力拒否(`param: input`)、またはルートが読み取れない入力メディアの拒否(`unsupported_input_modality`)を示す構造化エラーでは、出力開始前に次の適格なターゲットへ進めます。この不一致ではクールダウンを記録しません。安全ポリシーによる拒否、キャンセル、出力開始後の再実行は禁止のままです。 [Canonical compatibility details](/guides/combos/#request-local-target-compatibility). + +## クールダウン回復の予約 + +`cooldownReprobe: { enabled: true }` で有効にした回復試行は、予約したリクエストだけに属し、他のリクエストには保存済みのクールダウンが引き続き適用されます。予約はクールダウン待機と JEV 選択を経ても維持されます。待機中に再び利用可能になったターゲットは、クールダウンが変わっていなければ再試行できますが、新しいクールダウンの記録はそのターゲットの予約を無効にします。成功時は応答したターゲットの変更されていないクールダウンだけを解除し、未試行のターゲットや新しい失敗のクールダウンは残します。プロバイダーの無効化、クォータ、リクエストの適格性、キャンセルの確認は引き続き有効です。[設定と制限](/guides/combos/#retry-all-cooled-targets-opt-in)を参照してください。 diff --git a/docs-site/src/content/docs/ko/guides/combos.md b/docs-site/src/content/docs/ko/guides/combos.md index e5d8c844516..6abe4f0241f 100644 --- a/docs-site/src/content/docs/ko/guides/combos.md +++ b/docs-site/src/content/docs/ko/guides/combos.md @@ -289,3 +289,7 @@ opencodex 인스턴스에 기록했는지 확인하세요. 일반적인 400 오류는 종료되지만, `user`를 명시적으로 거부하거나 `reasoning.effort`/`reasoning_effort`의 지원되지 않는 값, 모델별 이미지 입력 거부(`param: input`) 또는 경로가 읽을 수 없는 입력 미디어 거부(`unsupported_input_modality`)를 나타내는 구조화된 오류는 출력 시작 전에 다음 적격 대상으로 넘어갈 수 있습니다. 이 경우 쿨다운을 기록하지 않습니다. 보안 정책 거부, 취소 및 이미 시작된 출력은 재실행하지 않습니다. [Canonical compatibility details](/guides/combos/#request-local-target-compatibility). + +## 쿨다운 복구 예약 + +`cooldownReprobe: { enabled: true }`로 활성화한 복구 시도는 이를 예약한 요청에만 속하며, 다른 요청에는 저장된 쿨다운이 계속 적용됩니다. 예약은 쿨다운 대기와 JEV 선택을 거쳐 유지됩니다. 대기 중 다시 사용 가능해진 대상은 기존 쿨다운이 바뀌지 않았다면 재시도할 수 있지만, 새 쿨다운 기록은 그 대상의 예약을 무효화합니다. 성공하면 응답한 대상의 변경되지 않은 쿨다운만 해제하며, 시도하지 않은 대상과 새로운 실패의 쿨다운은 유지합니다. 공급자 비활성화, 할당량, 요청 적격성 및 취소 검사는 계속 적용됩니다. [설정과 제한](/guides/combos/#retry-all-cooled-targets-opt-in)을 참고하세요. diff --git a/docs-site/src/content/docs/ru/guides/combos.md b/docs-site/src/content/docs/ru/guides/combos.md index 31b760bfd3d..b20b666a408 100644 --- a/docs-site/src/content/docs/ru/guides/combos.md +++ b/docs-site/src/content/docs/ru/guides/combos.md @@ -335,3 +335,7 @@ CLI и дашборд показывают точное сообщение ва Исключение для завершающих ошибок 400: структурированный отказ от `user`, неподдерживаемое значение `reasoning.effort`/`reasoning_effort`, специфичный для модели отказ входного изображения (`param: input`) или отказ маршрута, который не может прочитать входные медиаданные (`unsupported_input_modality`) позволяет до начала вывода перейти к следующей допустимой цели без периода охлаждения. Отказ политики безопасности, отмена и уже начавшийся вывод по-прежнему запрещают повторное выполнение. [Canonical compatibility details](/guides/combos/#request-local-target-compatibility). + +## Резервирование восстановления после cooldown + +Попытка восстановления, включённая через `cooldownReprobe: { enabled: true }`, принадлежит только зарезервировавшему её запросу; для других запросов сохранённые cooldown продолжают действовать. Резервирование сохраняется при ожидании cooldown и выборе JEV. Если цель снова становится доступной во время ожидания, её можно повторно попробовать, пока запись cooldown не изменилась. Новая запись отменяет резервирование этой цели. Успех снимает только неизменённый cooldown ответившей цели; для непроверенных целей и новых сбоев cooldown сохраняется. Проверки отключения провайдера, квоты, допустимости запроса и отмены остаются в силе. См. [настройки и ограничения](/guides/combos/#retry-all-cooled-targets-opt-in). diff --git a/docs-site/src/content/docs/tr/guides/combos.md b/docs-site/src/content/docs/tr/guides/combos.md index e2063712afc..44ab77f40b7 100644 --- a/docs-site/src/content/docs/tr/guides/combos.md +++ b/docs-site/src/content/docs/tr/guides/combos.md @@ -414,3 +414,7 @@ reddedilen istek kaynağını düzeltin. Kombolar bu durumlar için atlama yapma Sonlandırıcı 400 hatalarının dar bir istisnası vardır: `user` alanını açıkça reddeden, `reasoning.effort`/`reasoning_effort` için desteklenmeyen değer bildiren, modele özgü görüntü girdisini reddeden (`param: input`) veya rotanın okuyamadığı girdi medyasını reddeden (`unsupported_input_modality`) yapılandırılmış hata, çıktı başlamadan önce sonraki uygun hedefe geçebilir. Bu uyumsuzluk için bekleme süresi kaydedilmez. Güvenlik politikası reddi, iptal ve başlamış çıktı yeniden yürütülmez. [Canonical compatibility details](/guides/combos/#request-local-target-compatibility). + +## Soğuma kurtarma rezervasyonu + +`cooldownReprobe: { enabled: true }` ile etkinleştirilen kurtarma denemesi yalnızca onu ayıran isteğe aittir; diğer istekler kayıtlı soğuma sürelerine uymaya devam eder. Rezervasyon, soğuma beklemesi ve JEV seçimi boyunca korunur. Bekleme sırasında yeniden kullanılabilir olan hedef, soğuma kaydı değişmediyse tekrar denenebilir. Yeni bir soğuma kaydı o hedefin rezervasyonunu geçersiz kılar. Başarı yalnızca yanıt veren hedefin değişmemiş soğuma kaydını kaldırır; denenmeyen hedefler ve yeni hatalar soğuma kayıtlarını korur. Sağlayıcının devre dışı olması, kota, istek uygunluğu ve iptal kontrolleri geçerliliğini korur. [Ayarlar ve sınırlar](/guides/combos/#retry-all-cooled-targets-opt-in) bölümüne bakın. diff --git a/docs-site/src/content/docs/zh-cn/guides/combos.md b/docs-site/src/content/docs/zh-cn/guides/combos.md index efb8ddc85a5..8cb8bbe9e80 100644 --- a/docs-site/src/content/docs/zh-cn/guides/combos.md +++ b/docs-site/src/content/docs/zh-cn/guides/combos.md @@ -303,3 +303,7 @@ combo id 不存在。响应是 HTTP 404,类型为 `invalid_request_error`。 一般 400 错误仍会终止请求,但明确拒绝 `user`、对 `reasoning.effort`/`reasoning_effort` 返回不支持值、返回模型特定图像输入拒绝(`param: input`),或拒绝该路由无法读取的输入媒体(`unsupported_input_modality`)的结构化错误,可让 combo 在输出开始前尝试下一个符合条件的目标,而不记录冷却时间。安全策略拒绝、取消以及已经开始的输出仍不可重放。 [Canonical compatibility details](/guides/combos/#request-local-target-compatibility). + +## 冷却恢复预约 + +通过 `cooldownReprobe: { enabled: true }` 启用的恢复尝试只属于预约它的请求;其他请求仍受已保存的冷却限制。预约会在冷却等待和 JEV 选择期间保留。目标在等待期间重新可用时,只要其冷却记录没有变化,就仍可重试;新的冷却记录会使该目标的预约失效。成功后,只清除实际响应目标未改变的冷却记录;未尝试的目标和新的失败仍保留冷却。提供方禁用、配额、请求资格和取消检查仍然生效。请参阅[设置与限制](/guides/combos/#retry-all-cooled-targets-opt-in)。 diff --git a/docs-site/src/content/docs/zh-tw/guides/combos.md b/docs-site/src/content/docs/zh-tw/guides/combos.md index 86d26e0d0b9..5ff12338332 100644 --- a/docs-site/src/content/docs/zh-tw/guides/combos.md +++ b/docs-site/src/content/docs/zh-tw/guides/combos.md @@ -300,3 +300,7 @@ Combo id 未知。回應為 HTTP 404 並帶 type `invalid_request_error`。執 一般 400 錯誤仍會終止請求,但明確拒絕 `user`、對 `reasoning.effort`/`reasoning_effort` 回傳不支援值、回傳模型特定影像輸入拒絕(`param: input`),或拒絕該路由無法讀取的輸入媒體(`unsupported_input_modality`)的結構化錯誤,可讓 combo 在輸出開始前嘗試下一個符合條件的目標,而不記錄冷卻時間。安全政策拒絕、取消及已開始的輸出仍不可重播。 [Canonical compatibility details](/guides/combos/#request-local-target-compatibility). + +## 冷卻復原預約 + +透過 `cooldownReprobe: { enabled: true }` 啟用的復原嘗試只屬於預約它的請求;其他請求仍受已儲存的冷卻限制。預約會在冷卻等待與 JEV 選擇期間保留。目標在等待期間重新可用時,只要其冷卻紀錄沒有變更,就仍可重試;新的冷卻紀錄會使該目標的預約失效。成功後,只清除實際回應目標未變更的冷卻紀錄;未嘗試的目標與新的失敗仍保留冷卻。供應商停用、配額、請求資格與取消檢查仍然生效。請參閱[設定與限制](/guides/combos/#retry-all-cooled-targets-opt-in)。 diff --git a/src/combos/failover.ts b/src/combos/failover.ts index 1041f7ef037..3dda95285c5 100644 --- a/src/combos/failover.ts +++ b/src/combos/failover.ts @@ -13,6 +13,7 @@ import { interface TargetCooldown { cooldownUntil: number; + revision: number; status?: number; } @@ -44,6 +45,8 @@ const HTTP_MONTH_INDEX: Record = { /** Map<`${comboId}\0${provider/model}`, TargetCooldown> */ const targetCooldowns = new Map(); +// Never reset: an in-flight reservation may still refer to an entry cleared from the map. +let nextCooldownRevision = 0; let lastReconciledGeneration = 0; let liveComboTargets = new Set(); @@ -163,6 +166,42 @@ export function isComboTargetInCooldown( return true; } +/** The live cooldown expiry for one target, or undefined once it is absent or elapsed. */ +export function comboTargetCooldownExpiry( + comboId: string, + target: Pick, + now = Date.now(), +): number | undefined { + const key = cooldownMapKey(comboId, target); + const entry = targetCooldowns.get(key); + if (!entry) return undefined; + if (entry.cooldownUntil <= now) { + targetCooldowns.delete(key); + return undefined; + } + return entry.cooldownUntil; +} + +/** Identity of the current live write, independent of its possibly repeated expiry. */ +export function comboTargetCooldownRevision( + comboId: string, + target: Pick, + now = Date.now(), +): number | undefined { + return isComboTargetInCooldown(comboId, target, now) + ? targetCooldowns.get(cooldownMapKey(comboId, target))?.revision : undefined; +} + +/** Lift only the reserved write; a replacement with even the same expiry survives. */ +export function clearComboTargetCooldownIfUnchanged( + comboId: string, + target: Pick, + expectedRevision: number, +): void { + const key = cooldownMapKey(comboId, target); + if (targetCooldowns.get(key)?.revision === expectedRevision) targetCooldowns.delete(key); +} + export function isTransientRequestRateLimit(input: { status?: number; code?: string | null; @@ -262,6 +301,7 @@ export function coolComboTarget( message: options?.message, }) ? COMBO_REQUEST_RATE_COOLDOWN_MS : DEFAULT_COOLDOWN_MS); targetCooldowns.set(cooldownMapKey(comboId, target), { + revision: ++nextCooldownRevision, // Local fallbacks are capped at ten minutes; explicit server delays at one day. cooldownUntil: now + (serverDelayMs ?? Math.min(Math.max(cooldownMs, 1), MAX_COOLDOWN_MS)), status: options?.status, diff --git a/src/combos/index.ts b/src/combos/index.ts index d1a7d275a6f..bb6ec0c6bd1 100644 --- a/src/combos/index.ts +++ b/src/combos/index.ts @@ -30,6 +30,7 @@ export { pickComboTarget, pickComboTargetWithWait, quotaInactiveReason, + settleComboReprobeSuccess, snapshotComboQuotaCooldowns, tryPickComboModel, UnknownComboError, diff --git a/src/combos/resolve.ts b/src/combos/resolve.ts index 60d07e6ba61..4886d7371fb 100644 --- a/src/combos/resolve.ts +++ b/src/combos/resolve.ts @@ -3,7 +3,8 @@ import { getCachedProviderRoutingQuota } from "../providers/quota-routing-cache" import type { ProviderQuota, ProviderQuotaWindow } from "../providers/quota-types"; import { sleepWithAbort } from "../lib/upstream-retry"; import { - clearComboTargetCooldowns, + clearComboTargetCooldownIfUnchanged, + comboTargetCooldownRevision, coolComboTarget, earliestComboCooldown, isComboTargetInCooldown, @@ -24,6 +25,8 @@ export interface ComboPick { targetIndex: number; attempted: string[]; writerGeneration: number; + /** Cooled targets this request alone may probe, each bound to the cooldown write it reserved. */ + cooldownReprobeTargets?: Record; } interface SelectionState { @@ -288,11 +291,11 @@ function resetWindowIndex( return selected; } -/** Reserve one recovery pass before mutating cooldowns; preview and in-flight picks never call this. */ -function reopenAllCooledTargets( +/** Reserve one request-scoped recovery pass; preview and in-flight picks never call this. */ +function reserveAllCooledTargets( config: OcxConfig, comboId: string, now: number, eligible?: (target: NormalizedComboTarget) => boolean, -): void { +): Map | undefined { const combo = getCombo(config, comboId); if (!combo?.cooldownReprobe?.enabled) return; const state = selectionState.get(comboId); @@ -307,13 +310,22 @@ function reopenAllCooledTargets( if (combo.targets.some(target => !isComboTargetInCooldown(comboId, target, now) && reprobeEligible(target))) return; const usable = combo.targets.filter(reprobeEligible); - if (usable.length === 0 || !usable.every(target => isComboTargetInCooldown(comboId, target, now))) return; + if (usable.length === 0) return; + // Each reserved key is bound to the cooldown entry observed here. A replacement entry — + // a provider-scoped failure's, or one this request's own failed hop wrote — is newer + // evidence and cancels that target's bypass instead of being probed through. + const reserved = new Map(); + for (const target of usable) { + const revision = comboTargetCooldownRevision(comboId, target, now); + if (revision === undefined) return; + reserved.set(targetKey(target), revision); + } // No await between reservation and reset: concurrent requests cannot reserve another pass. const next = state ?? { successes: 0, currentWeights: new Map(), successfulUses: new Map() }; next.lastCooldownReprobeAt = now; selectionState.set(comboId, next); - clearComboTargetCooldowns(comboId, usable.map(targetKey)); - console.warn(`[combo] ${comboId}: reopening all-cooled targets; minimum reprobe interval ${combo.cooldownReprobe.intervalMs}ms`); + console.warn(`[combo] ${comboId}: reserving all-cooled targets for one request; minimum reprobe interval ${combo.cooldownReprobe.intervalMs}ms`); + return reserved; } export function pickComboTarget( @@ -324,6 +336,8 @@ export function pickComboTarget( eligible?: (target: NormalizedComboTarget) => boolean; /** Disable recovery for picks without request context or picks already in flight. */ allowCooldownReprobe?: boolean; + /** Request-scoped cooldown bypasses, each bound to the write it was reserved against. */ + cooldownReprobeTargets?: Iterable; now?: number; /** Inspect a round-robin choice without mutating its sticky/weight state. */ preview?: boolean; @@ -334,12 +348,19 @@ export function pickComboTarget( if (!combo) throw new UnknownComboError(comboId); const excluded = new Set(options.exclude ?? []); const now = options.now ?? Date.now(); + const cooldownReprobeTargets = new Map(options.cooldownReprobeTargets); if (options.allowCooldownReprobe !== false && !options.preview && excluded.size === 0) { - reopenAllCooledTargets(config, comboId, now, options.eligible); + for (const [key, revision] of reserveAllCooledTargets(config, comboId, now, options.eligible) ?? []) { + cooldownReprobeTargets.set(key, revision); + } } + const reprobeBypass = (target: NormalizedComboTarget): boolean => { + const revision = cooldownReprobeTargets.get(targetKey(target)); + return revision !== undefined && revision === comboTargetCooldownRevision(comboId, target, now); + }; const eligible = (target: NormalizedComboTarget): boolean => targetProviderIsUsable(config, target, now) - && !isComboTargetInCooldown(comboId, target, now) + && (reprobeBypass(target) || !isComboTargetInCooldown(comboId, target, now)) && !excluded.has(targetKey(target)) && (options.eligible?.(target) ?? true); @@ -417,6 +438,7 @@ export function pickComboTarget( targetIndex, attempted: [...excluded, targetKey(target)], writerGeneration, + ...(cooldownReprobeTargets.size > 0 ? { cooldownReprobeTargets: Object.fromEntries(cooldownReprobeTargets) } : {}), }; } @@ -447,6 +469,17 @@ export function noteComboSuccess( } } +/** + * A request that proved a reserved target healthy lifts exactly the cooldown entry it + * probed; a replacement entry written while the request was in flight stays. Targets + * the pass reserved but never dispatched keep their cooldowns. + */ +export function settleComboReprobeSuccess(pick: ComboPick): void { + const revision = pick.cooldownReprobeTargets?.[targetKey(pick.target)]; + if (revision === undefined) return; + clearComboTargetCooldownIfUnchanged(pick.comboId, pick.target, revision); +} + export function noteComboFailure( comboId: string, target: OcxComboTarget, @@ -519,8 +552,10 @@ export function advanceComboAfterFailure( return pickComboTarget(config, pick.comboId, { exclude: pick.attempted, now: options.now, - eligible: target => !isComboTargetInCooldown(pick.comboId, target, options.now) - && (options.eligible?.(target) ?? true) + cooldownReprobeTargets: pick.cooldownReprobeTargets + ? Object.entries(pick.cooldownReprobeTargets) + : undefined, + eligible: target => (options.eligible?.(target) ?? true) && (!defersLastResort || !target.lastResort), }); } @@ -535,6 +570,8 @@ export async function pickComboTargetWithWait( abortSignal?: AbortSignal; /** Disable recovery when retrying a target after its normal cooldown. */ allowCooldownReprobe?: boolean; + /** Request-scoped cooldown bypasses carried through the wait, each bound to its write. */ + cooldownReprobeTargets?: Iterable; now?: number; sleep?: (ms: number, signal?: AbortSignal) => Promise; }, @@ -542,13 +579,12 @@ export async function pickComboTargetWithWait( const now = options.now ?? Date.now(); const excluded = new Set(options.exclude ?? []); const customEligible = options.eligible; + const cooldownReprobeTargets = new Map(options.cooldownReprobeTargets); if (options.allowCooldownReprobe !== false && excluded.size === 0 && !options.abortSignal?.aborted) { - reopenAllCooledTargets(config, comboId, now, customEligible); + for (const [key, revision] of reserveAllCooledTargets(config, comboId, now, customEligible) ?? []) { + cooldownReprobeTargets.set(key, revision); + } } - const eligibleAt = (target: NormalizedComboTarget, at: number): boolean => - !isComboTargetInCooldown(comboId, target, at) - && (customEligible?.(target) ?? true); - const eligible = (target: NormalizedComboTarget): boolean => eligibleAt(target, now); // Milliseconds already slept inside this call. `waitForCooldownMs` is documented as a cap // per *selection attempt*, so a deferral wait and the ordinary wait below must share it — // otherwise a 3s deferral followed by a 9s ordinary wait spends 12s against a 10s budget. @@ -567,9 +603,10 @@ export async function pickComboTargetWithWait( && policyCombo.targets.some(target => !target.lastResort); if (defersLastResort && !options.abortSignal?.aborted) { const normalOnly = (target: NormalizedComboTarget): boolean => - !target.lastResort && eligible(target); + !target.lastResort && (customEligible?.(target) ?? true); const normalPick = pickComboTarget(config, comboId, { allowCooldownReprobe: false, + cooldownReprobeTargets, exclude: excluded, eligible: normalOnly, now, @@ -601,11 +638,10 @@ export async function pickComboTargetWithWait( if (!getCombo(config, comboId)) return null; const waited = pickComboTarget(config, comboId, { allowCooldownReprobe: false, + cooldownReprobeTargets, exclude: excluded, now: now + normalDelay, - eligible: target => !target.lastResort - && !isComboTargetInCooldown(comboId, target, now + normalDelay) - && (customEligible?.(target) ?? true), + eligible: normalOnly, }); if (waited) return waited; } @@ -619,8 +655,9 @@ export async function pickComboTargetWithWait( const pick = pickComboTarget(config, comboId, { allowCooldownReprobe: false, + cooldownReprobeTargets, exclude: excluded, - eligible: target => eligibleAt(target, clock), + eligible: customEligible, now: clock, }); if (pick || remainingWaitMs <= 0 || options.abortSignal?.aborted) return pick; @@ -654,8 +691,9 @@ export async function pickComboTargetWithWait( return pickComboTarget(config, comboId, { exclude: excluded, allowCooldownReprobe: false, + cooldownReprobeTargets, now: clock + delay, - eligible: targetCandidate => eligibleAt(targetCandidate, clock + delay), + eligible: customEligible, }); } diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index 64e296ade5f..ad8aa6c7a6b 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -35,6 +35,7 @@ import { comboExclusionReasons, comboTargetSkipTrigger, noteComboSuccess, + settleComboReprobeSuccess, comboFailureDecision, advanceComboAfterFailure, comboFailureCooldownScope, @@ -319,19 +320,23 @@ interface JevComboChoice { candidate: JevCandidate; } -/** Enumerate the current ordinary Combo eligibility set without retaining attempted picks. */ +/** Enumerate this request's eligible targets without reserving another recovery pass. */ function eligibleJevComboChoices( config: OcxConfig, comboId: string, eligible: (target: NonNullable>["targets"][number]) => boolean, now: number, + cooldownReprobeTargets?: ComboPick["cooldownReprobeTargets"], ): JevComboChoice[] { const combo = getCombo(config, comboId); if (!combo) return []; const excluded = new Set(); const choices: JevComboChoice[] = []; + const reserved = cooldownReprobeTargets ? Object.entries(cooldownReprobeTargets) : undefined; while (excluded.size < combo.targets.length) { - const pick = pickComboTarget(config, comboId, { exclude: excluded, eligible, now }); + const pick = pickComboTarget(config, comboId, { + exclude: excluded, eligible, now, allowCooldownReprobe: false, cooldownReprobeTargets: reserved, + }); if (!pick) break; const key = targetKey(pick.target); excluded.add(key); @@ -582,6 +587,8 @@ export async function executeComboResponses( exclude?: Iterable; eligible?: (target: NonNullable["targets"][number]) => boolean; allowCooldownReprobe?: boolean; + /** This request's own cooldown-reprobe reservation, carried through its failover hops. */ + cooldownReprobeTargets?: Iterable; now?: number; }) => pickComboTargetWithWait(config, comboId, { ...pickOptions, @@ -672,22 +679,20 @@ export async function executeComboResponses( } let jevDecision: JevDecision | undefined; if (combo.strategy === "jev" && !toolLoopPin) { - const choices = eligibleJevComboChoices(config, comboId, targetEligible, Date.now()); + const choices = eligibleJevComboChoices(config, comboId, targetEligible, Date.now(), pick.cooldownReprobeTargets); const first = choices[0]; if (!first) { return comboUnavailable(comboId, undefined, comboExclusionReasons(config, comboId, target => targetEligible(target) && target.provider !== JEV_PROVIDER_ID && jevChoiceEfforts(config, target, false) !== undefined)); } - const resolvedFailOpenEffort = resolveEffortAtOrBelow( - "medium", - first.candidate.reasoningEfforts, - ); - const fallback: Pick = { - targetKey: first.candidate.key, - effort: resolvedFailOpenEffort && isCodexReasoningEffort(resolvedFailOpenEffort) - ? resolvedFailOpenEffort as OcxComboDefaultEffort - : null, + const fallbackFor = (choice: JevComboChoice): Pick => { + const effort = resolveEffortAtOrBelow("medium", choice.candidate.reasoningEfforts); + return { + targetKey: choice.candidate.key, + effort: effort && isCodexReasoningEffort(effort) ? effort as OcxComboDefaultEffort : null, + }; }; + const fallback = fallbackFor(first); const decisionStartedAt = Date.now(); const pinReuse = jevPin?.reuse( choices.map(choice => choice.candidate), @@ -729,9 +734,32 @@ export async function executeComboResponses( latencyMs: Math.max(0, Date.now() - decisionStartedAt), }; } + let selected = choices.find(choice => choice.candidate.key === decision.targetKey) ?? first; + let selectionUnavailable = false; + if (pick.cooldownReprobeTargets && !options.abortSignal?.aborted) { + // Another request can replace a cooldown while the decision is awaited. Recheck the + // original reservation, current availability and request eligibility without reserving again. + const currentChoices = eligibleJevComboChoices(config, comboId, targetEligible, Date.now(), pick.cooldownReprobeTargets); + const current = currentChoices.find(choice => choice.candidate.key === decision.targetKey + && (decision.effort === null ? choice.candidate.reasoningEfforts.length === 0 + : choice.candidate.reasoningEfforts.includes(decision.effort))); + const available = current ?? currentChoices[0]; + if (available) selected = available; + else selectionUnavailable = true; + if (!current) { + // Keep completed decision usage even when no target remains. Old scores and quota + // selection describe neither a fail-open replacement nor a rejected selection. + decision = { + backend: decision.backend, + ...(available ? fallbackFor(available) : { targetKey: decision.targetKey, effort: decision.effort }), + gate: decision.gate === "apply" ? "invalid" : decision.gate, + latencyMs: decision.latencyMs, + ...(decision.usage ? { usage: decision.usage } : {}), + }; + } + } if (pinReuse?.released) decision = { ...decision, pinReason: pinReuse.released }; jevDecision = decision; - const selected = choices.find(choice => choice.candidate.key === decision.targetKey) ?? first; pick = { ...selected.pick, attempted: [targetKey(selected.pick.target)] }; logCtx.jevDecision = normalizePersistedJevDecision({ version: 1, @@ -766,6 +794,11 @@ export async function executeComboResponses( : {}), ...(decision.usage ? { usage: decision.usage } : {}), }); + // A completed decision belongs to the parent log even if no model can be dispatched. + if (selectionUnavailable) { + return comboUnavailable(comboId, undefined, comboExclusionReasons(config, comboId, target => + targetEligible(target) && target.provider !== JEV_PROVIDER_ID && jevChoiceEfforts(config, target, false) !== undefined)); + } } // One immutable combo selection trace, before any child dispatch; child // adoption below must never replace it with a concrete child route trace. @@ -1170,6 +1203,7 @@ export async function executeComboResponses( ); attemptRetained = true; noteComboSuccess(comboId, combo, pick.target, pick.writerGeneration); + settleComboReprobeSuccess(pick); Object.assign(logCtx, childLog, { requestedModel, model: requestedModel, @@ -1368,10 +1402,13 @@ export async function executeComboResponses( && failedTargetCooldownRecorded && isComboTargetInCooldown(comboId, pick.target, failureNow); // Drop exclusions for the single-target retry, but wait for normal cooldown - // expiry: an in-flight request cannot reserve a fresh recovery pass. - const retryAfterCooldown = () => + // expiry: an in-flight request cannot reserve a fresh recovery pass, though the + // reservation it already owns still applies to entries unchanged since it was made. + const carriedReprobe: ComboPick["cooldownReprobeTargets"] = pick.cooldownReprobeTargets; + const retryAfterCooldown = (): Promise => pickWithWait({ allowCooldownReprobe: false, + cooldownReprobeTargets: carriedReprobe ? Object.entries(carriedReprobe) : undefined, eligible: targetEligible, now: failureNow, }); @@ -1381,6 +1418,7 @@ export async function executeComboResponses( pick = await pickWithWait({ exclude: pick.attempted, eligible: targetEligible, + cooldownReprobeTargets: carriedReprobe ? Object.entries(carriedReprobe) : undefined, now: failureNow, }); // A single-target combo with waitForCooldownMs has no alternate target to fail over to, @@ -1414,6 +1452,7 @@ export async function executeComboResponses( const recoveredTarget = await pickWithWait({ exclude: attemptedTargets, eligible: recoveryEligible, + cooldownReprobeTargets: carriedReprobe ? Object.entries(carriedReprobe) : undefined, now: failureNow, }); if (recoveredTarget && await recoverUnreadableEncryptedTask()) { diff --git a/structure/fallback-policy.md b/structure/fallback-policy.md index ac3c1f0ff10..f8465a63fa3 100644 --- a/structure/fallback-policy.md +++ b/structure/fallback-policy.md @@ -130,4 +130,4 @@ privacy), `tests/responses/responses-combo-fallback-off.test.ts`, `tests/routing `tests/server/fallback-routes.test.ts`, `tests/clients/client-home-unreachable.test.ts`, `tests/cli/cli-fallback.test.ts` and `gui/tests/fallback-panel.test.tsx`. -The opt-in [all-cooled Combo recovery](runtime.md#scoped-provider-quota-for-combo-selection) can reopen saved combo cooldowns in `src/combos/resolve.ts`; only the request-eligible target keys are cleared, with throwing predicates treated as ineligible during recovery preparation and available siblings preferred without probing cooled targets. Route-level picks and in-flight retry-after-cooldown selections disable reprobes. Recovery does not authorize a fallback hop or bypass the configured fallback policy. +The opt-in [all-cooled Combo recovery](runtime.md#scoped-provider-quota-for-combo-selection) can reserve saved combo cooldowns in `src/combos/resolve.ts`; only that request bypasses its reserved writes until a successful dispatch conditionally clears its unchanged target, with throwing predicates treated as ineligible during recovery preparation and available siblings preferred without probing cooled targets. Route-level picks and in-flight retry-after-cooldown selections disable reprobes. Recovery does not authorize a fallback hop or bypass the configured fallback policy. diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index 09fd2a3a145..2ac67762689 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -594,6 +594,6 @@ Account-row projection is loaded with `import()` after each account probe in `sr remains authoritative. No raw body is retained in that field, serialized into client errors, or logged. Presence of both `detail` and `error` across the original root and its own non-array `response` record is ambiguous, regardless of values; null counts and deeper quoted objects do not. This is checked before selecting an error carrier. For complete bounded HTTP errors, the text wrapper selects root `detail`/`error` when present; otherwise it selects the own `response` record for the same exact `detail` or `error.message` match. It never searches another carrier after a nonmatch or descends further. Policy-fallback and direct callers share this wrapper; SSE message parsing disables nested HTTP selection. The same bounded HTTP parser retains allowlisted dedicated `code` fields from root and response before display truncation; diagnostic `type` is never promoted, including when code is absent or malformed. Policy/direct classification inspects full text only without authoritative carrier metadata; SSE never borrows hard codes from quoted message JSON. -Committed output, cancellation, hard policy refusals and resend limits remain stronger than model fallback. Opt-in all-cooled target recovery in `src/combos/resolve.ts` follows [the Combo cooldown contract](runtime.md#scoped-provider-quota-for-combo-selection) without widening account/provider admission. It clears only request-eligible target cooldowns; route-level selection (including an unmetered reroute) and in-flight cooldown retries cannot initiate a reprobe. Encrypted-task recovery preserves request-specific JEV, reasoning-replay and protocol eligibility while relaxing payload readability. +Committed output, cancellation, hard policy refusals and resend limits remain stronger than model fallback. Opt-in all-cooled target recovery in `src/combos/resolve.ts` follows [the Combo cooldown contract](runtime.md#scoped-provider-quota-for-combo-selection) without widening account/provider admission. It reserves only request-eligible target cooldown writes for that request; a successful dispatch clears only its unchanged reserved write, while shared cooldowns exclude other requests; route-level selection (including an unmetered reroute) and in-flight cooldown retries cannot initiate a reprobe. Encrypted-task recovery preserves request-specific JEV, reasoning-replay and protocol eligibility while relaxing payload readability. Token Plan dated-model policy (2026-10-20): both curated presets include exact `deepseek-v4.1-flash-1017` metadata without changing live discovery or Responses pins. The registry may seed `noStructuredOutputModels` at route resolution, unioned with operator opt-outs, just as it seeds `noJsonSchemaModels`. The full opt-out wins on both translated and native Chat paths. For 1017 this deliberately omits JSON requests: strict schemas are rejected and a `json_object` fallback has not been verified. The bare model's independently verified downgrade remains separate. Token Plan startup refreshes only exact previous registry rosters for both presets, adding dated models and removing retired International IDs. International rows refresh only on the Token Plan host or the registry default; pay-as-you-go and Custom destinations keep their lists. Edited or reordered lists, saved model defaults and existing per-model window, output-limit and effort overrides remain untouched; an added ID is backfilled only into such a map the row already has, and a row already on the current roster gets only its missing output limits, because custom catalog rows read the saved output-limit map directly. Destination-based enrichment of custom-named rows fills per-model effort defaults, effort maps and structured-output restrictions from the matching registry destination or alias, and operator values win. Both Token Plan presets keep the hosts their official quick starts document; International custom-name enrichment remains excluded by its existing override guard. diff --git a/structure/providers/jev-decision.md b/structure/providers/jev-decision.md index a6d2b06b2f7..368e343f10f 100644 --- a/structure/providers/jev-decision.md +++ b/structure/providers/jev-decision.md @@ -207,6 +207,14 @@ applicable decision, so operators must keep secrets and private paths out of it. An absent note leaves the prior decision payload shape intact. `src/server/responses/core-combo.ts` computes current eligibility and asks JEV once for the initial pick. +Candidate enumeration carries the initial pick's request-scoped cooldown reservation and cannot +reserve another pass. After the decision returns, reserved requests recheck current eligibility and +cooldown revisions before dispatch. A stale choice fails open to the first remaining eligible target +and its supported effort, recording that actual fallback. No remaining choice returns unavailable +without sending; the completed decision keeps its reported usage and zero model attempts. A previously +applicable choice is marked invalid; existing failure gates remain. The selected pick retains the reservation through ordinary failover and +conditional success settlement under the [Combo cooldown contract](../runtime.md#scoped-provider-quota-for-combo-selection). +`tests/server/server-combo-cooldown-fallback.test.ts` covers an all-cooled JEV choice, its fallback and concurrent reservation invalidation. The validated effort shapes the first Responses child or the separate opt-in native Chat body in `src/server/responses/core-combo-native.ts`; both remove caller `service_tier` and conflicting effort controls. Explicit null strips effort using a non-force path, even when the Combo has a forced default; the ordinary diff --git a/structure/runtime.md b/structure/runtime.md index 9c540d419b7..8ea4fe63bc3 100644 --- a/structure/runtime.md +++ b/structure/runtime.md @@ -504,7 +504,7 @@ This is also why the classifier cannot duplicate visible output. Native byte str Regression coverage: `tests/responses/responses-forward-prompt-envelope.test.ts`, `tests/routing/router-combo-failover-classification.test.ts`, `tests/routing/routing-policy-fallback.test.ts`, `tests/helpers/combo-context-overflow-cases.ts`, and `tests/server/server-combo-failover-e2e.test.ts`. Every combo, routing-profile, subagent and hard-lock hop to another model or provider first passes the [fallback policy](fallback-policy.md#sites). -`src/combos/failover.ts` uses a 10-minute fallback for a spent account usage window (codes `usage_limit_exceeded`, `usage_limit_reached`, `1308`, or `usage limit reached` / `usage limit has been reached` / `token-plan quota has been exhausted` prose, including HTTP 502) and for provider-scoped credential or billing failure codes such as `invalid_api_key` and `insufficient_quota`. This duration does not change failure classification or cooldown scope; upstream retry/reset signals and configured durations retain precedence. It caps explicit upstream `Retry-After` target cooldowns at 24 hours while reset-derived, configured, and fallback cooldowns remain capped at 10 minutes. `src/combos/resolve.ts` omits a target cooldown for a 429 only when the failed child identifies one account in an Anthropic OAuth or canonical Codex pool and that account has a live pool cooldown, or identifies one account in a generic OAuth pool that still has an eligible sibling. `src/server/responses/core-combo.ts` checks the pool's health record before passing that attribution; for a generic OAuth pool it first hands the account (matched through its `accountLogLabel`) to `rotateGenericOAuthAccountOn429`, which cools it under the pool's own Retry-After policy even when the per-request account budget ended rotation before that account was recorded, and attributes the failure only when the rotator names a sibling. Unknown accounts, ambiguous labels, a lone or fully cooled generic OAuth pool, API-key providers, and other failures keep their ordinary target cooldown. An API-key pool needs no attribution: its 429 rotation walks every key inside the attempt, so a 429 reaches the combo only once every key is cooled. Regression coverage: `tests/oauth/generic-oauth-combo-account-cooldown.test.ts`. A 402 is the provider-scoped twin for an Anthropic OAuth pool: the pool rotates only on 401/403/429, so when the failed child identifies one pool account and the failure carries the exact credit-balance refusal (`isAnthropicBillingFailure`), `core-combo.ts` records that account's refusal through the pool's `recordAnthropicAccountRefusal` (the same entitlement backoff a 403 credit refusal gets) and `resolve.ts` omits every target cooldown only while the pool still has an eligible account; a lone account, an exhausted pool, an unidentified account, or any other 402 keeps the provider-scoped cooldown. Regression coverage: `tests/adapters/anthropic/anthropic-combo-account-cooldown.test.ts`. The Anthropic OAuth pool's own all-accounts-cooled 429 is the one unattributed exception: `isAnthropicPoolLocalRefusal` in `core-combo.ts` requires the `anthropic` provider with `authMode: "oauth"`, status 429, no identified failed account, no eligible pool account and a live pool retry-after. The target then still cools, but for the local fallback instead of the restated per-account `Retry-After`, so a resumed or newly added account is not ignored until a 24-hour target cooldown expires. An upstream 429 on an API-key provider or on an identified pool account keeps its full `Retry-After`. Regression coverage: `tests/routing/always-on-429-failover.test.ts`. Cooldown entries in `src/combos/failover.ts` retain the failure status. The quota-only snapshot query records target/expiry pairs for unexpired 429/402 cooldowns using only cooldown map lookups. Its config-aware wrapper in `src/combos/resolve.ts` applies the picker's provider availability and cached quota checks before dispatch; request eligibility runs lazily only when exhaustion considers a 503, with throwing checks treated as ineligible. Responses exhaustion uses this evidence under the [failover contract](transports/responses-failover.md); ordinary cooldown selection and timing stay unchanged. Opt-in `cooldownReprobe: { enabled: true, intervalMs }` in `src/combos/resolve.ts` permits an initial non-preview request to clear all-cooled eligible targets once per configured interval (default 300000 ms, range 1000..86400000). The synchronous reservation lives in selection state and resets on proxy restart or selection-state reset; Only target keys eligible for the current request are cleared; other target cooldowns are preserved. Route-level selection without request eligibility disables reprobes, as does the in-flight retry-after-cooldown path, which waits for normal expiry. Nested wait-path selections cannot start a second reprobe. Preparation first checks available siblings without evaluating cooled targets; a throwing eligibility predicate is ineligible for recovery. Encrypted-task recovery relaxes only payload readability and retains metering, JEV, reasoning-replay and protocol-lane eligibility. Its recovery selection may reserve a reprobe only after Responses ingress, thread-spawn, enabled recovery configuration and non-child-attempt admission pass; refused recovery preserves cooldowns and the reservation. New failures record cooldowns normally, and provider quota, account admission, payload eligibility, metered-spend and cancellation checks remain authoritative. There is no timer; existing send bounds remain. The opt-in may retry a target before its server Retry-After. CLI `--cooldown-reprobe on|off` and `--cooldown-reprobe-interval` control it; omitted management fields preserve it on unrelated saves. Tests: `tests/codex-integration/combo-cooldown-reprobe.test.ts` and `tests/cli/cli-combo-parity.test.ts`. A zero-attempt `combo_unavailable` names why each target was skipped. `comboExclusionReasons` in `src/combos/resolve.ts` derives one coarse label per target from the picker's own checks, in picker order: `provider missing`, `provider disabled`, `quota exhausted`, `cooldown s`, then `ineligible for this request` when the caller's eligibility predicate refuses it or throws. `NoAvailableComboTargetsError` and `comboUnavailable` append them to the message; the label carries only the configured provider/model, never account, credential or request data. Coverage: `tests/responses/responses-combo-exhausted-error.test.ts`. +`src/combos/failover.ts` uses a 10-minute fallback for a spent account usage window (codes `usage_limit_exceeded`, `usage_limit_reached`, `1308`, or `usage limit reached` / `usage limit has been reached` / `token-plan quota has been exhausted` prose, including HTTP 502) and for provider-scoped credential or billing failure codes such as `invalid_api_key` and `insufficient_quota`. This duration does not change failure classification or cooldown scope; upstream retry/reset signals and configured durations retain precedence. It caps explicit upstream `Retry-After` target cooldowns at 24 hours while reset-derived, configured, and fallback cooldowns remain capped at 10 minutes. `src/combos/resolve.ts` omits a target cooldown for a 429 only when the failed child identifies one account in an Anthropic OAuth or canonical Codex pool and that account has a live pool cooldown, or identifies one account in a generic OAuth pool that still has an eligible sibling. `src/server/responses/core-combo.ts` checks the pool's health record before passing that attribution; for a generic OAuth pool it first hands the account (matched through its `accountLogLabel`) to `rotateGenericOAuthAccountOn429`, which cools it under the pool's own Retry-After policy even when the per-request account budget ended rotation before that account was recorded, and attributes the failure only when the rotator names a sibling. Unknown accounts, ambiguous labels, a lone or fully cooled generic OAuth pool, API-key providers, and other failures keep their ordinary target cooldown. An API-key pool needs no attribution: its 429 rotation walks every key inside the attempt, so a 429 reaches the combo only once every key is cooled. Regression coverage: `tests/oauth/generic-oauth-combo-account-cooldown.test.ts`. A 402 is the provider-scoped twin for an Anthropic OAuth pool: the pool rotates only on 401/403/429, so when the failed child identifies one pool account and the failure carries the exact credit-balance refusal (`isAnthropicBillingFailure`), `core-combo.ts` records that account's refusal through the pool's `recordAnthropicAccountRefusal` (the same entitlement backoff a 403 credit refusal gets) and `resolve.ts` omits every target cooldown only while the pool still has an eligible account; a lone account, an exhausted pool, an unidentified account, or any other 402 keeps the provider-scoped cooldown. Regression coverage: `tests/adapters/anthropic/anthropic-combo-account-cooldown.test.ts`. The Anthropic OAuth pool's own all-accounts-cooled 429 is the one unattributed exception: `isAnthropicPoolLocalRefusal` in `core-combo.ts` requires the `anthropic` provider with `authMode: "oauth"`, status 429, no identified failed account, no eligible pool account and a live pool retry-after. The target then still cools, but for the local fallback instead of the restated per-account `Retry-After`, so a resumed or newly added account is not ignored until a 24-hour target cooldown expires. An upstream 429 on an API-key provider or on an identified pool account keeps its full `Retry-After`. Regression coverage: `tests/routing/always-on-429-failover.test.ts`. Cooldown entries in `src/combos/failover.ts` retain the failure status. The quota-only snapshot query records target/expiry pairs for unexpired 429/402 cooldowns using only cooldown map lookups. Its config-aware wrapper in `src/combos/resolve.ts` applies the picker's provider availability and cached quota checks before dispatch; request eligibility runs lazily only when exhaustion considers a 503, with throwing checks treated as ineligible. Responses exhaustion uses this evidence under the [failover contract](transports/responses-failover.md); ordinary cooldown selection and timing stay unchanged. Opt-in `cooldownReprobe: { enabled: true, intervalMs }` in `src/combos/resolve.ts` permits an initial non-preview request to reserve all-cooled eligible targets once per configured interval (default 300000 ms, range 1000..86400000). The synchronous reservation lives in selection state and resets on proxy restart or selection-state reset; the reserved target keys are carried only by that request's picks; shared cooldowns remain recorded and continue to exclude concurrent requests. Each reservation names the cooldown write's unique revision, not its expiry. A successful dispatch settles through `settleComboReprobeSuccess`, which lifts only that unchanged reserved write; unprobed and re-cooled targets stay recorded, including replacements with an identical Retry-After expiry. Route-level selection without request eligibility disables reprobes, as does the in-flight retry-after-cooldown path, which waits for normal expiry. Nested wait-path and JEV selections retain the same reservation without starting a second reprobe. After ordinary or last-resort-deferral waits and after an awaited JEV decision, selection rechecks provider and request eligibility at the current clock and permits only unchanged-revision bypasses. Preparation first checks available siblings without evaluating cooled targets; a throwing eligibility predicate is ineligible for recovery. Encrypted-task recovery relaxes only payload readability and retains metering, JEV, reasoning-replay and protocol-lane eligibility. Its recovery selection may reserve a reprobe only after Responses ingress, thread-spawn, enabled recovery configuration and non-child-attempt admission pass; refused recovery preserves cooldowns and the reservation. New failures record cooldowns normally, and provider quota, account admission, payload eligibility, metered-spend and cancellation checks remain authoritative. There is no timer; existing send bounds remain. The opt-in may retry a target before its server Retry-After. CLI `--cooldown-reprobe on|off` and `--cooldown-reprobe-interval` control it; omitted management fields preserve it on unrelated saves. Tests: `tests/codex-integration/combo-cooldown-reprobe.test.ts`, `tests/server/server-combo-cooldown-fallback.test.ts` and `tests/cli/cli-combo-parity.test.ts`. A zero-attempt `combo_unavailable` names why each target was skipped. `comboExclusionReasons` in `src/combos/resolve.ts` derives one coarse label per target from the picker's own checks, in picker order: `provider missing`, `provider disabled`, `quota exhausted`, `cooldown s`, then `ineligible for this request` when the caller's eligibility predicate refuses it or throws. `NoAvailableComboTargetsError` and `comboUnavailable` append them to the message; the label carries only the configured provider/model, never account, credential or request data. Coverage: `tests/responses/responses-combo-exhausted-error.test.ts`. ## Combo default effort precedence diff --git a/tests/cli/cli-codex-cli-update-boundary.test.ts b/tests/cli/cli-codex-cli-update-boundary.test.ts index c0c82b7bb5f..05600eae972 100644 --- a/tests/cli/cli-codex-cli-update-boundary.test.ts +++ b/tests/cli/cli-codex-cli-update-boundary.test.ts @@ -119,6 +119,7 @@ describe("Codex CLI selection inspection boundary", () => { // chain, end to end against a real npm install -g of a synthetic fixture // package into an owned temporary prefix. Only the registry answer is stubbed // (a fake npm first on the bound PATH); the install itself is real npm's work. + // Packing, installing, and inspecting this fixture can exceed Bun's default 5s. test.skipIf(process.platform === "win32")("plan identity binds the bytes a real npm-global install placed", () => { const node = Bun.which("node"); const npm = Bun.which("npm"); @@ -343,7 +344,7 @@ describe("Codex CLI selection inspection boundary", () => { const removedPlan = JSON.parse(removed.stdout); expect(removedPlan.applicable).toBe(false); expect(removedPlan.refusal).toBe("install_content_unverified"); - }); + }, 60_000); }); function inspectionContext(transportEnv?: unknown, extra: Record = {}) { diff --git a/tests/codex-integration/combo-cooldown-reprobe.test.ts b/tests/codex-integration/combo-cooldown-reprobe.test.ts index 03284daeb6a..ee265edb3c7 100644 --- a/tests/codex-integration/combo-cooldown-reprobe.test.ts +++ b/tests/codex-integration/combo-cooldown-reprobe.test.ts @@ -1,6 +1,6 @@ import { describe, test, expect } from 'bun:test'; -import { pickComboTarget, pickComboTargetWithWait, advanceComboAfterFailure, clearComboSelectionState } from '../../src/combos/resolve'; -import { clearComboTargetCooldowns, coolComboTarget, isComboTargetInCooldown } from '../../src/combos/failover'; +import { pickComboTarget, pickComboTargetWithWait, advanceComboAfterFailure, clearComboSelectionState, settleComboReprobeSuccess } from '../../src/combos/resolve'; +import { clearComboTargetCooldowns, comboTargetCooldownRevision, coolComboTarget, isComboTargetInCooldown } from '../../src/combos/failover'; import { comboConfigIssues, normalizeComboConfig } from '../../src/combos/types'; import { routeModel } from '../../src/router'; import type { OcxConfig, OcxComboConfig } from '../../src/types'; @@ -55,8 +55,7 @@ describe('opt-in cooldown recovery', () => { }); test('available targets prevent a reset of siblings', () => { const { id, config } = setup(); - pickComboTarget(config, id, { now }); - coolComboTarget(id, targets[0]!, { now, retryAfter: '36000', status: 402 }); + clearComboTargetCooldowns(id, ['openai/test-b']); expect(pickComboTarget(config, id, { now })!.target.model).toBe('test-b'); expect(isComboTargetInCooldown(id, targets[0]!, now)).toBe(true); }); @@ -64,9 +63,67 @@ describe('opt-in cooldown recovery', () => { const { id, config } = setup(); const pick = pickComboTarget(config, id, { now, eligible: target => target.model === 'test-a' }); expect(pick!.target.model).toBe('test-a'); - expect(isComboTargetInCooldown(id, targets[0]!, now)).toBe(false); + expect(isComboTargetInCooldown(id, targets[0]!, now)).toBe(true); expect(isComboTargetInCooldown(id, targets[1]!, now)).toBe(true); }); + test('a reserved pass does not make cooled targets visible to concurrent requests', () => { + const { id, config } = setup(); + const reserved = pickComboTarget(config, id, { now })!; + expect(reserved.target.model).toBe('test-a'); + expect(isComboTargetInCooldown(id, targets[0]!, now)).toBe(true); + expect(pickComboTarget(config, id, { now })).toBeNull(); + const second = advanceComboAfterFailure(config, reserved, { + now, status: 429, retryAfter: '36000', cooldownScope: 'target', + }); + expect(second!.target.model).toBe('test-b'); + expect(pickComboTarget(config, id, { now })).toBeNull(); + }); + test('a failed single-target reprobe cannot bypass the same absolute Retry-After again', async () => { + const { id, config } = setup(); + config.combos![id]!.targets = [targets[0]!]; + const retryAfter = new Date(now + 30_000).toUTCString(); + coolComboTarget(id, targets[0]!, { now, retryAfter, status: 429 }); + const reserved = pickComboTarget(config, id, { now })!; + expect(advanceComboAfterFailure(config, reserved, { + now: now + 1_000, status: 429, retryAfter, cooldownScope: 'target', + })).toBeNull(); + const sleeps: number[] = []; + const retried = await pickComboTargetWithWait(config, id, { + now: now + 1_000, waitForCooldownMs: 1_000, allowCooldownReprobe: false, + cooldownReprobeTargets: Object.entries(reserved.cooldownReprobeTargets!), + sleep: async delay => { sleeps.push(delay); }, + }); + expect(retried).toBeNull(); + expect(sleeps).toEqual([]); + expect(isComboTargetInCooldown(id, targets[0]!, now + 1_000)).toBe(true); + }); + test('a provider failure replaces same-expiry reservations for every sibling', () => { + const { id, config } = setup(); + const retryAfter = new Date(now + 30_000).toUTCString(); + for (const target of targets) coolComboTarget(id, target, { now, retryAfter, status: 429 }); + const reserved = pickComboTarget(config, id, { now })!; + expect(advanceComboAfterFailure(config, reserved, { + now: now + 1_000, status: 429, retryAfter, cooldownScope: 'provider', + })).toBeNull(); + expect(targets.every(target => isComboTargetInCooldown(id, target, now + 1_000))).toBe(true); + }); + test('successful reprobe settlement preserves a replacement with the same expiry', () => { + const { id, config } = setup(); + const reserved = pickComboTarget(config, id, { now })!; + // A concurrent failure can write identical expiry and status, even in the same tick. + coolComboTarget(id, reserved.target, { now, retryAfter: '36000', status: 429 }); + settleComboReprobeSuccess(reserved); + expect(isComboTargetInCooldown(id, reserved.target, now)).toBe(true); + expect(pickComboTarget(config, id, { now })).toBeNull(); + }); + test('successful reprobe settlement releases only the unchanged dispatched target', () => { + const { id, config } = setup(); + const reserved = pickComboTarget(config, id, { now })!; + settleComboReprobeSuccess(reserved); + expect(isComboTargetInCooldown(id, reserved.target, now)).toBe(false); + expect(isComboTargetInCooldown(id, targets[1]!, now)).toBe(true); + expect(pickComboTarget(config, id, { now })!.target.model).toBe('test-a'); + }); for (const wait of [false, true]) { test(`a cooled throwing predicate cannot block an available sibling (wait=${wait})`, async () => { const { id, config } = setup(); @@ -130,6 +187,62 @@ describe('opt-in cooldown recovery', () => { expect(sleeps).toEqual([1000]); expect(pick!.target.model).toBe('test-a'); }); + for (const policy of [undefined, 'before-last-resort'] as const) { + for (const afterWait of ['usable', 'disabled', 'ineligible', 'recooled'] as const) { + test(`wait rechecks a reserved target's current eligibility (${policy ?? 'ordinary'}, ${afterWait})`, async () => { + const id = `reprobe-wait-${++sequence}`; + const waitedTargets = ['a', 'b', 'c', ...(policy ? ['d'] : [])].map(provider => ({ + provider, model: `test-${provider}`, lastResort: provider === 'd', + })); + const config = { + providers: Object.fromEntries(waitedTargets.map(target => [target.provider, { + baseUrl: 'https://example.invalid', apiKey: 'test', + }])), + combos: { [id]: { strategy: 'failover', targets: waitedTargets, + cooldownReprobe: { enabled: true }, ...(policy ? { cooldownWaitPolicy: policy } : {}) } }, + } as OcxConfig; + for (const target of waitedTargets) coolComboTarget(id, target, { now, retryAfter: '36000', status: 429 }); + const first = (await pickComboTargetWithWait(config, id, { now, waitForCooldownMs: 1000 }))!; + expect(first.target.provider).toBe('a'); + const reservedB = first.cooldownReprobeTargets!['b/test-b']; + expect(reservedB).toBe(comboTargetCooldownRevision(id, waitedTargets[1]!, now)); + + // While A is in flight, B is disabled and C gets a newer, shorter cooldown. + // C's replacement invalidates its reservation and makes a real wait necessary. + config.providers.b!.disabled = true; + coolComboTarget(id, waitedTargets[2]!, { now, retryAfter: '1', status: 429 }); + expect(advanceComboAfterFailure(config, first, { + now, status: 429, retryAfter: '36000', cooldownScope: 'target', + })).toBeNull(); + let bEligible = true; + const sleeps: number[] = []; + const pick = await pickComboTargetWithWait(config, id, { + now, waitForCooldownMs: 1000, allowCooldownReprobe: false, + exclude: first.attempted, cooldownReprobeTargets: Object.entries(first.cooldownReprobeTargets!), + eligible: target => target.provider !== 'b' || bEligible, + sleep: async delay => { + sleeps.push(delay); + expect(comboTargetCooldownRevision(id, waitedTargets[1]!, now)).toBe(reservedB); + config.providers.b!.disabled = afterWait === 'disabled'; + bEligible = afterWait !== 'ineligible'; + if (afterWait === 'recooled') { + coolComboTarget(id, waitedTargets[1]!, { now, retryAfter: '36000', status: 429 }); + } + }, + }); + expect(sleeps).toEqual([1000]); + expect(pick!.target.provider).toBe(afterWait === 'usable' ? 'b' : 'c'); + expect(isComboTargetInCooldown(id, waitedTargets[1]!, now + 1000)).toBe(true); + if (afterWait === 'usable') { + expect(pick!.cooldownReprobeTargets!['b/test-b']).toBe(reservedB); + expect(comboTargetCooldownRevision(id, waitedTargets[1]!, now + 1000)).toBe(reservedB); + settleComboReprobeSuccess(pick!); + expect(isComboTargetInCooldown(id, waitedTargets[1]!, now + 1000)).toBe(false); + expect(isComboTargetInCooldown(id, waitedTargets[0]!, now + 1000)).toBe(true); + } + }); + } + } test('wait path reserves synchronously across concurrent callers', async () => { const { id, config } = setup(); const first = await pickComboTargetWithWait(config, id, { now, waitForCooldownMs: 0 }); diff --git a/tests/providers/github-copilot/github-copilot-wire-defaults.test.ts b/tests/providers/github-copilot/github-copilot-wire-defaults.test.ts index 3c6ddf4d0de..036699fe0ea 100644 --- a/tests/providers/github-copilot/github-copilot-wire-defaults.test.ts +++ b/tests/providers/github-copilot/github-copilot-wire-defaults.test.ts @@ -12,12 +12,14 @@ import { afterEach, describe, expect, spyOn, test } from "bun:test"; import * as oauth from "../../../src/oauth"; import { fetchProviderModels } from "../../../src/codex/catalog/provider-fetch"; import { clearModelCache } from "../../../src/codex/model-cache"; +import { clearCopilotAutoModelsForTests } from "../../../src/providers/github-copilot-auto"; import { providerConfigSeed } from "../../../src/providers/derive"; import { getProviderRegistryEntry } from "../../../src/providers/registry"; import { resolveWireProtocolOverride } from "../../../src/server/adapter-resolve"; import { handleResponses } from "../../../src/server/responses/core"; import type { OcxConfig, OcxProviderConfig } from "../../../src/types"; import { acquireOwnedSpendHome } from "../../helpers/owned-spend-home"; +import { withStubbedProviderFetch } from "../../helpers/catalog-provider-fetch"; let releaseSpendHome: (() => void) | undefined; @@ -151,27 +153,42 @@ describe("the wire default survives the handleResponses replay", () => { // Release the preload-home lease before later teardown can replace or remove that home. releaseSpendHome?.(); releaseSpendHome = undefined; + clearCopilotAutoModelsForTests(); globalThis.fetch = originalFetch; }); - function captureUpstreamUrl(): string[] { - const urls: string[] = []; - globalThis.fetch = (async (input: RequestInfo | URL) => { - urls.push(String(input)); + function captureUpstreamUrls(): { catalogUrls: string[]; inferenceUrls: string[] } { + const catalogUrls: string[] = []; + const inferenceUrls: string[] = []; + globalThis.fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const url = input instanceof Request ? input.url : String(input); + const method = init?.method ?? (input instanceof Request ? input.method : "GET"); + if (url === "https://api.githubcopilot.com/models") { + expect(method).toBe("GET"); + catalogUrls.push(url); + // Named-model permission discovery precedes inference and expects JSON. + return Response.json({ data: [...RESPONSES_ONLY, ...CHAT_SERVED] + .map(id => ({ id, model_picker_enabled: true })) }); + } + expect(method).toBe("POST"); + inferenceUrls.push(url); return new Response("data: [DONE]\n\n", { status: 200, headers: { "content-type": "text/event-stream" }, }); }) as typeof fetch; - return urls; + return { catalogUrls, inferenceUrls }; } async function drive(model: string, inboundWire?: "responses" | "chat" | "anthropic"): Promise { - const urls = captureUpstreamUrl(); - const config = { providers: { "github-copilot": copilotProvider() } } as unknown as OcxConfig; + clearCopilotAutoModelsForTests(); + const { catalogUrls, inferenceUrls } = captureUpstreamUrls(); + const config = withStubbedProviderFetch({ + providers: { "github-copilot": copilotProvider() }, + }) as unknown as OcxConfig; // Direct dispatch needs the writer lease that startServer normally owns for this home. releaseSpendHome = acquireOwnedSpendHome(); - await handleResponses( + const response = await handleResponses( new Request("http://localhost/v1/responses", { method: "POST", headers: { "content-type": "application/json" }, @@ -183,7 +200,14 @@ describe("the wire default survives the handleResponses replay", () => { { model: "", provider: "" }, inboundWire === undefined ? {} : { inboundWire }, ); - return urls[0] ?? ""; + try { + expect(response.status).toBe(200); + expect(catalogUrls).toEqual(["https://api.githubcopilot.com/models"]); + expect(inferenceUrls).toHaveLength(1); + return inferenceUrls[0] ?? ""; + } finally { + await response.body?.cancel(); + } } test("gpt-5.4 reaches the Responses endpoint, never /chat/completions", async () => { diff --git a/tests/responses/responses-pool-401-refresh.test.ts b/tests/responses/responses-pool-401-refresh.test.ts index 4e1d10b50b6..f869ba8a002 100644 --- a/tests/responses/responses-pool-401-refresh.test.ts +++ b/tests/responses/responses-pool-401-refresh.test.ts @@ -91,6 +91,8 @@ function config(options: { secondAccount?: boolean } = {}): OcxConfig { baseUrl: "https://chatgpt.com/backend-api/codex", authMode: "forward", codexAccountMode: "pool", + // This fixture intercepts HTTP fetch; no test may dial a live WebSocket. + upstreamWebsocket: false, }, }, codexAccounts: options.secondAccount @@ -283,6 +285,7 @@ function encryptedRecoveryComboConfig(options: { adapter: "openai-responses", baseUrl: "https://chatgpt.com/backend-api/codex", authMode: "forward", + upstreamWebsocket: false, }; } if (options.extraSpare) { diff --git a/tests/server/server-combo-cooldown-fallback.test.ts b/tests/server/server-combo-cooldown-fallback.test.ts index 76cabb070c0..848a35aa4b2 100644 --- a/tests/server/server-combo-cooldown-fallback.test.ts +++ b/tests/server/server-combo-cooldown-fallback.test.ts @@ -7,7 +7,8 @@ import { comboProviderFactory } from "../helpers/combo-provider"; import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; import { removeTreeWithRetry } from "../helpers/remove-tree"; -import { clearComboSelectionState, clearComboTargetCooldowns } from "../../src/combos"; +import { clearComboSelectionState, clearComboTargetCooldowns, coolComboTarget, isComboTargetInCooldown } from "../../src/combos"; +import { comboTargetCooldownExpiry, comboTargetCooldownRevision } from "../../src/combos/failover"; import { clearComboRecallForTests } from "../../src/server/responses/combo-session-recall"; import { clearKeyCooldowns } from "../../src/providers/key-failover"; import { clearCodexUpstreamHealth } from "../../src/codex/routing"; @@ -128,6 +129,225 @@ function serve(handler: (request: Request) => Response | Promise) { } describe("combo cooldown-ready fallback", () => { + test.each([false, true])("JEV carries an all-cooled reservation through decision and fallback (failed choice=%s)", async failedChoice => { + const hits: string[] = []; + const first = serve(() => { + hits.push("a"); + return Response.json(responsesSuccess("reserved first target")); + }); + const selected = serve(() => { + hits.push("b"); + return failedChoice + ? Response.json({ error: { message: "fixture limit", type: "rate_limit_error" } }, + { status: 429, headers: { "retry-after": "600" } }) + : Response.json(responsesSuccess("reserved JEV choice")); + }); + const decisions: Array<{ questions: { route: { criteria: Record } } }> = []; + const config = comboConfig({ + a: provider("openai-responses", baseUrl(first), "key-a", { + liveModels: false, models: ["m1"], modelReasoningEfforts: { m1: [] }, + }), + b: provider("openai-responses", baseUrl(selected), "key-b", { + liveModels: false, models: ["m2"], modelReasoningEfforts: { m2: [] }, + }), + }, undefined, { strategy: "jev", cooldownReprobe: { enabled: true } }); + config.providers.jev = provider("jev-decision", "https://api.typesafe.ai/v1/systemone", "jev-fixture-key", { + liveModels: false, + fetch: (async (_input, init) => { + decisions.push(JSON.parse(String(init?.body))); + return Response.json({ answers: { route: { choice: "b/m2:none", confidence: 0.8 } } }); + }) as typeof fetch, + }); + for (const target of config.combos!.free!.targets) coolComboTarget("free", target, { retryAfter: "600", status: 429 }); + const logCtx: RequestLogContext = { model: "", provider: "" }; + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "combo/free", input: "hello", stream: false }), + }), config, logCtx); + expect(response.status).toBe(200); + expect(await response.text()).toContain(failedChoice ? "reserved first target" : "reserved JEV choice"); + expect(decisions).toHaveLength(1); + expect(Object.keys(decisions[0]!.questions.route.criteria)).toEqual(["a/m1:none", "b/m2:none"]); + expect(logCtx.jevDecision?.gate).toBe("apply"); + expect(logCtx.jevDecision?.selected).toMatchObject({ provider: "b", model: "m2" }); + expect(hits).toEqual(failedChoice ? ["b", "a"] : ["b"]); + expect(isComboTargetInCooldown("free", config.combos!.free!.targets[0]!)).toBe(!failedChoice); + expect(isComboTargetInCooldown("free", config.combos!.free!.targets[1]!)).toBe(failedChoice); + }); + + test.each([false, true])("JEV rejects a reservation replaced while its decision is pending (all replaced=%s)", async replaceAll => { + const entered = Promise.withResolvers(); + const finish = Promise.withResolvers(); + const hits: string[] = []; + const bodies: Array> = []; + const first = serve(async request => { + hits.push("a"); + bodies.push(await request.json() as Record); + return Response.json(responsesSuccess("current reserved fallback")); + }); + const selected = serve(() => { + hits.push("b"); + return Response.json(responsesSuccess("stale reserved choice")); + }); + const config = comboConfig({ + a: provider("openai-responses", baseUrl(first), "key-a", { + liveModels: false, models: ["m1"], modelReasoningEfforts: { m1: ["low"] }, + }), + b: provider("openai-responses", baseUrl(selected), "key-b", { + liveModels: false, models: ["m2"], modelReasoningEfforts: { m2: ["high"] }, + }), + }, undefined, { strategy: "jev", reasoningEffortMode: "adaptive", cooldownReprobe: { enabled: true } }); + const decisions: Array<{ questions: { route: { criteria: Record } } }> = []; + config.providers.jev = provider("jev-decision", "https://api.typesafe.ai/v1/systemone", "jev-fixture-key", { + liveModels: false, + fetch: (async (_input, init) => { + decisions.push(JSON.parse(String(init?.body))); + entered.resolve(); + await finish.promise; + return Response.json({ + answers: { route: { choice: "b/m2:high", confidence: 0.8 } }, + usage: { input_tokens: 3, output_tokens: 1 }, + }); + }) as typeof fetch, + }); + const targets = config.combos!.free!.targets; + const retryAfter = new Date(Date.now() + 60_000).toUTCString(); + for (const target of targets) coolComboTarget("free", target, { retryAfter, status: 429 }); + const originalRevision = comboTargetCooldownRevision("free", targets[1]!); + const originalExpiry = comboTargetCooldownExpiry("free", targets[1]!); + const logCtx: RequestLogContext = { model: "", provider: "" }; + const pending = handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "combo/free", input: "hello", stream: false }), + }), config, logCtx); + let barrierValidated = false; + try { + await Promise.race([entered.promise, pending.then(() => { + throw new Error("request completed before the JEV decision barrier"); + })]); + // A concurrent failure writes new evidence while JEV is deciding. Matching expiry + // must not make this request's older reservation own the replacement cooldown. + for (const target of replaceAll ? targets : [targets[1]!]) { + coolComboTarget("free", target, { retryAfter, status: 429 }); + } + expect(comboTargetCooldownRevision("free", targets[1]!)).not.toBe(originalRevision); + expect(comboTargetCooldownExpiry("free", targets[1]!)).toBe(originalExpiry); + barrierValidated = true; + } finally { + finish.resolve(); + if (!barrierValidated) { + // Join the request before fixture teardown without replacing the original assertion. + try { await (await pending).body?.cancel(); } catch { /* Preserve the barrier failure. */ } + } + } + const response = await pending; + const text = await response.text(); + expect(hits).toEqual(replaceAll ? [] : ["a"]); + expect(response.status).toBe(replaceAll ? 503 : 200); + expect(text).toContain(replaceAll ? "combo_unavailable" : "current reserved fallback"); + expect(decisions).toHaveLength(1); + expect(Object.keys(decisions[0]!.questions.route.criteria)).toEqual(["a/m1:low", "b/m2:high"]); + expect(isComboTargetInCooldown("free", targets[1]!)).toBe(true); + expect(isComboTargetInCooldown("free", targets[0]!)).toBe(replaceAll); + expect(logCtx.jevDecision).toMatchObject({ + selected: replaceAll ? { provider: "b", model: "m2", effort: "high" } + : { provider: "a", model: "m1", effort: "low" }, + gate: "invalid", backend: "typesafe", latencyMs: expect.any(Number), + usage: { inputTokens: 3, outputTokens: 1, totalTokens: 4 }, + }); + expect(logCtx.jevDecision?.confidence).toBeUndefined(); + expect(logCtx.jevDecision?.chosenProbability).toBeUndefined(); + if (replaceAll) expect((logCtx.attempts ?? []).reduce((total, attempt) => total + attempt.sendCount, 0)).toBe(0); + else expect(bodies[0]!.reasoning).toMatchObject({ effort: "low" }); + }); + + test("successful reprobe stays exclusive until dispatch succeeds, then releases its target", async () => { + const started = Promise.withResolvers(); + const finish = Promise.withResolvers(); + let hits = 0; + const upstream = serve(async () => { + hits++; + started.resolve(); + await finish.promise; + return Response.json(responsesSuccess("reprobe recovered")); + }); + const target = { provider: "a", model: "m1" }; + const config = comboConfig({ a: provider("openai-responses", baseUrl(upstream), "key-a") }, [target], + { cooldownReprobe: { enabled: true } }); + coolComboTarget("free", target, { retryAfter: "600", status: 429 }); + const post = () => handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "combo/free", input: "hello", stream: false }), + }), config, { model: "", provider: "" }); + const pending = post(); + try { + await started.promise; + const concurrent = await post(); + expect(concurrent.status).toBe(503); + await concurrent.text(); + expect(hits).toBe(1); + expect(isComboTargetInCooldown("free", target)).toBe(true); + } finally { finish.resolve(); } + const recovered = await pending; + expect(recovered.status).toBe(200); + expect(await recovered.text()).toContain("reprobe recovered"); + expect(isComboTargetInCooldown("free", target)).toBe(false); + const later = await post(); + expect(later.status).toBe(200); + await later.text(); + expect(hits).toBe(2); + }); + + test("failed single-target reprobe cannot send twice through an identical Retry-After date", async () => { + const retryAfter = new Date(Date.now() + 60_000).toUTCString(); + let hits = 0; + const upstream = serve(() => { + hits++; + return Response.json({ error: { message: "fixture limit", type: "rate_limit_error" } }, + { status: 429, headers: { "retry-after": retryAfter } }); + }); + const target = { provider: "a", model: "m1" }; + const config = comboConfig({ a: provider("openai-chat", baseUrl(upstream), "key-a") }, [target], + { cooldownReprobe: { enabled: true }, waitForCooldownMs: 1 }); + coolComboTarget("free", target, { retryAfter, status: 429 }); + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "combo/free", input: "hello", stream: false }), + }), config, { model: "", provider: "" }); + expect(response.status).toBe(429); + await response.text(); + expect(hits).toBe(1); + expect(isComboTargetInCooldown("free", target)).toBe(true); + }); + + test("a reserved last-resort target survives the server's wait-policy fallback", async () => { + let firstHits = 0; + let backupHits = 0; + customRunTurn = async (_parsed, _incoming, emit) => { + firstHits++; + emit({ type: "error", status: 429, errorType: "rate_limit_error", message: "fixture limit" }); + }; + const backup = serve(() => { + backupHits++; + return Response.json(responsesSuccess("reserved backup recovered")); + }); + const config = comboConfig({ + a: provider("test-run-turn", "test://run-turn", "key-a"), + b: provider("openai-responses", baseUrl(backup), "key-b"), + }, undefined, { cooldownReprobe: { enabled: true }, cooldownWaitPolicy: "before-last-resort", waitForCooldownMs: 0 }); + config.combos!.free!.targets[1]!.lastResort = true; + for (const target of config.combos!.free!.targets) coolComboTarget("free", target, { retryAfter: "600", status: 429 }); + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "combo/free", input: "hello", stream: false }), + }), config, { model: "", provider: "" }); + expect(response.status).toBe(200); + expect(await response.text()).toContain("reserved backup recovered"); + expect([firstHits, backupHits]).toEqual([1, 1]); + expect(isComboTargetInCooldown("free", config.combos!.free!.targets[0]!)).toBe(true); + expect(isComboTargetInCooldown("free", config.combos!.free!.targets[1]!)).toBe(false); + }); + test("runTurn cooldown-ready heartbeat preserves combo fallback on a final 429", async () => { let firstHits = 0; let backupHits = 0;