fix(streaming): retry Responses before semantic output

[yinkev]

Summary

• keep OpenAI Responses SSE prefixes and provisional lifecycle events inside the bounded synchronous bootstrap window;
• treat response.queued, response.created, and response.in_progress as retryable even after their complete JSON payload arrives;
• commit on the first output-bearing event, terminal event, [DONE], or unknown event;
• discard buffered chunks from a failed attempt and replay only the successful attempt in order.

Refs #4527.

Root cause

Streaming bootstrap originally treated every non-empty translated chunk as the point of no return. The first fix kept split event: prefixes retryable, but a complete provisional frame still committed because any non-empty data: payload was considered semantic:

event: response.created
data: {"type":"response.created", ...}

If the upstream disconnected after that frame but before output or a terminal event, the client had still received no usable response content, yet streaming.bootstrap-retries was disabled.

The commitment rule is now event-aware only for the OpenAI Responses protocol. Provisional lifecycle events remain buffered under the existing 32-chunk / 64 KiB cap. All other protocols retain their existing first-deliverable-payload behavior. Broader recovery after semantic output remains outside this PR.

Tests

Added coverage for:

1. provisional lifecycle classification for raw JSON and SSE data: frames;
2. output-bearing, terminal, [DONE], and unknown events committing the response;
3. attempt 1 emitting a complete response.created frame and then a 408-style error;
4. attempt 2 succeeding, with only its buffered lifecycle and terminal frames reaching downstream;
5. the existing bounded prefix buffer and split-SSE behavior.

Validation:

go test ./sdk/api/handlers/... ./sdk/cliproxy/auth -count=1
go test -race ./sdk/api/handlers -run 'Test(StreamBootstrapPayloadCommitsResponse_OpenAIResponsesLifecycle|ExecuteStreamWithAuthManager_(RetriesAfterOpenAIResponsesProvisionalLifecycleFrame|BoundsOpenAIResponsesPrefixBuffer|AllowsSplitOpenAIResponsesSSEEventLines|RetriesAfterDroppedBootstrapPayload))$' -count=1
go vet ./sdk/api/handlers/... ./sdk/cliproxy/auth
go build -o test-output ./cmd/server && rm test-output
git diff --check

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 35505dc4b1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[zzl360]

基于最新代码，合并了这个PR，跑了5个小时，未再见这个错误

[fashen97]

HTTP 408
stream error: stream disconnected before completion: stream closed before response.completed
合并PR后仍遇到408问题，不过出现频率好像低了很多

[jroth1111]

Related complete recovery work is now available in #4643.

#4580 safely retains syntactic event:/comment prefixes during bootstrap. #4643 keeps that legacy low-latency behavior compatible, and adds an opt-in stronger mode that withholds the complete translated attempt through terminal success. That allows response.created / response.in_progress / response.queued, semantic text/reasoning/tool output, and a later terminal server_error to be discarded together before retry, with no losing bytes reaching the client.

The new mode is bounded by attempts, retry-start window, bytes, and concurrent recoveries, and fails open to ordinary streaming on overflow/saturation. Maintainers can therefore keep #4580 as the minimal default-path fix or use #4643 for the stronger delayed-response guarantee.

[zzl360]

@yinkev We reproduced an adjacent pre-semantic failure after integrating this PR locally.

The missing case is a complete provisional lifecycle frame, rather than only the split event: prefix:

event: response.created
data: {"type":"response.created", ...}
<stream closes before any semantic output or terminal event>

streamBootstrapPayloadCommitsResponse currently treats the non-empty data: payload as commitment, so bootstrap retry no longer applies even though no text, reasoning, tool call, or other semantic output has reached the downstream client.

This also reproduced after the client had successfully compacted the session to 24,987 input tokens (about a 96 KiB request body): four attempts returned HTTP 200 with zero usage and then closed before response.completed. That rules out context pressure as the sole explanation for this case.

Would it make sense for #4580 to keep only provisional Responses lifecycle events (response.queued, response.created, and response.in_progress) inside the existing bounded bootstrap buffer, and commit on the first semantic output or terminal event? A regression could make attempt 1 emit event: + complete data: response.created + 408-style stream error, then make attempt 2 complete successfully and assert that only attempt 2 reaches downstream.

This is narrower than the full-stream opt-in recovery in #4643: it never replays after semantic output, so it preserves the duplicate-output safety boundary described in #4527 while allowing downstream users to carry a single focused PR.

[yinkev]

Yes, that was the remaining hole. I pushed bf72dd3f.

response.queued, response.created, and response.in_progress now stay inside the bounded bootstrap window even after their complete JSON payload arrives. The stream commits on the first output-bearing event, any terminal event, [DONE], or an unknown event.

The regression now matches your reproduction: attempt 1 emits a complete response.created frame and then a 408; attempt 2 succeeds; only attempt 2 reaches downstream. The handler/auth suites, focused race run, vet, build, and diff check all pass.

I kept #4643 separate because this PR still stops strictly at the pre-semantic retry boundary rather than adding full-attempt recovery.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 032f5c8bda

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7b28c7ac64

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 07aaac58be

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[zzl360]

补充一个生产环境中的剩余 case：我们本地集成了这个 PR，并扩展到缓存 response.queued、response.created、response.in_progress 后，预语义阶段的断流明显减少，但语义输出开始后的断流仍持续出现。

2026-08-01 最近一小时的统计：

• gpt-5.6-sol：477 次执行，20 次失败尝试，对应 19 个最终未恢复请求；
• gpt-5.6-terra：527 次执行，6 个最终未恢复请求；
• 这些请求在 CPA HTTP 日志中全部是 200，但 usage 为 0 token，随后在 response.completed 前关闭；
• 没有对应的 4xx/5xx、context_too_large 或结构化上游错误；
• 失败分布在多个客户端会话和多个 credential，不是单一 session/key。

这说明 #4580 覆盖的“首个语义输出前”重试边界是有效的，但还不能处理首个文本、reasoning 或 tool 输出已经提交后发生的 incomplete EOF。这个 case 无法安全地在当前低延迟路径中直接重放，否则可能重复输出或重复工具调用；它更接近 #4643 的完整 attempt 缓冲、丢弃失败 attempt、只释放成功 attempt 的恢复模型。

建议至少在本 PR 的范围说明或回归说明中保留这个边界，并确保后续 #4643 覆盖：首个语义事件之后、response.completed 之前发生 EOF，失败 attempt 不应到达下游，重试成功后只释放 winning attempt。