Skip to content

fix(responses): recover cold opaque state and harden replay, archive and lifecycle paths - #7388

Merged
lidge-jun merged 10 commits into
devfrom
fix/responses-replay-lifecycle-2.90
Oct 19, 2026
Merged

lidge-jun merged 10 commits into
devfrom
fix/responses-replay-lifecycle-2.90

Conversation

@lidge-jun

Copy link
Copy Markdown
Owner

Summary

Existing threads stopped working after a model or provider switch (#7369, a 2.88.0 regression). The upstream rejected the replayed encrypted reasoning inside an HTTP-200 SSE stream. Recovery only ran when the request carried the process-local retention marker. A thread restored after a restart or switch has no marker, so it never recovered and every later turn failed.

With this change, the bounded zero-output preflight runs whenever the outbound body actually carries opaque state. A flat SSE type:error event that carries the dedicated invalid_encrypted_content code, or the exact unverifiable-ciphertext wording, now counts as a rejection. A 502 that carries the same rejection identity is accepted under the existing strict classifier. One sanitized rebuild follows a proven pre-output rejection. A repeated rejection is terminal, a generic 502 still authorizes nothing, and a native turn that succeeds keeps its ciphertext.

The bundle also carries five more fixes on the same replay and lifecycle paths:

  • Chat reasoning replay budgets are checked before copying.
  • Aggregate reasoning archive storage is capped.
  • Forward base URLs are normalized with a linear scan.
  • A discarded attempt's reasoning stays out of the retry's item.
  • A tracked stream ends when its turn controller is aborted.

Items fixed

Closes #7369

Carried PRs

PR Author What was taken Adapted / dropped
#7383 @Ingwannu Whole head bcefb120a: cold-history recovery in core-opaque-recovery.ts and combo-stream-preflight.ts, ADR-7369, native-reasoning-retention.md, responses-cold-opaque-recovery.test.ts, the Azure test tweak, layout maps Taken unchanged
#7334 @luvs01 Both commits: replay budget checks before copying (openai-chat/reasoning-replay.ts, openai-chat/passthrough.ts, chat-native.ts), structure docs, scope test Taken unchanged
#7309 @luvs01 Both commits: aggregate archive cap (reasoning-retention.ts), reasoning-archive-error.ts, the HTTP mapping in serve-options.ts, quota and HTTP tests, docs-site server config Taken unchanged. Extended with one follow-up commit; see the review finding below
#7348 @luvs01 Linear-scan base-URL normalization in openai-responses/transport.ts plus its test Taken unchanged
#7331 @vadymhimself Discarded-attempt reasoning separation in empty-completion-guard.ts plus its test Taken unchanged; usage merging and the no-post-output-replay controls are untouched
#7335 @vadymhimself Both commits: end a tracked stream on turn-controller abort in src/server/lifecycle.ts, plus a shutdown-drain test Taken unchanged. src/server/lifecycle.ts is an owner-only CODEOWNERS path. Exactly-once cleanup, pending-read cancellation and settled-lease tests in tests/service/shutdown-drain.test.ts stay green

All carried heads cherry-picked onto 79fa97be6 with no conflicts. Original authorship is kept on every carried commit.

Follow-up from review

A sol (gpt-6.1-sol) review of the full diff found no P0 or P1 issues. Its one P2 is fixed in 88473ed41. #7309 added the sanitized archive 503 at the HTTP boundary only. Over WebSocket, response.create hit the raw catch in src/server/index/websocket-handler.ts and sent a 502 proxy_error whose message included the absolute archive path and lacked the stable code. That catch now uses the same reasoningArchiveErrorResponse mapping and sends a 503 error frame. The new tests/responses/reasoning-archive-ws.test.ts drives a real proxy WebSocket through quota, lock-busy and unsafe-directory refusals. It asserts a 503 frame with the stable code, no path or secret text, zero upstream sends and an unchanged archive directory. Before the fix it failed (received 502). It is registered in layout.json and test-layout-expected.json, and structure/transports/responses-failover.md names it.

Judgment calls

  • I carried every head unchanged. Each one applied cleanly and passed its focused tests on current dev, so rewriting them would only add review surface.
  • I fixed the reviewer's P2 even though only P0/P1 fixes were required. It is a path disclosure on the same refusal the fix(responses): cap aggregate reasoning archive storage #7309 HTTP path already sanitizes, and the fix is ten lines.

Held out

Verification

All on this branch at 88473ed41, rebased on origin/dev 79fa97be6 (dev had not moved at push time).

  • bun run typecheck: pass.
  • Focused suites with bun test --isolate: 417 pass / 0 fail across 23 files. The files cover chat-inbound/native reasoning replay and scope, chat-native-spend, empty-completion core/guard/hardening, opaque-blob wrapped rejection, protocol opaque state, reasoning-archive-http, reasoning-retention, -quota, -routing and -v2, Azure, cold and standard opaque-blob recovery, retained-reasoning stream recovery, forward-transport base URL, tests/service/shutdown-drain.test.ts, tests/lab/core-lab-boundary.test.ts, and both test-layout tests.
  • tests/responses/reasoning-archive-ws.test.ts: 3/3 with the fix and 0/3 with the fix stashed (it received 502). Layout, ratchet and structure tests after registration: 87/87.
  • fix(responses): cap aggregate reasoning archive storage #7309 maintainer-requested evidence, all in that 417-pass run:
    • reasoning-retention-quota.test.ts: cross-process lock contention (a lock held by another process fails busy, writes nothing, and the retry archives, which runs a child process), over-cap legacy directory preserved, scan and close failure priority, non-regular, symlinked and uninspectable entries refusing closed.
    • reasoning-archive-http.test.ts: the production router's quota, busy, I/O and partial-write refusals with zero sends, for v1 and v2 and for stream and non-stream.
    • Windows-specific ACL and lock limits are covered only by CI-skipped cases on this macOS host.
  • bun run test:changed: it resolved to 1531 files. 34155 pass / 42 skip / 1 fail. The one failure, stage audio: audio buffer and audio response frames cross the relay unchanged in tests/server/server-live-realtime-fixtures.test.ts, is a timing flake: it passed 8/8 three times alone. The host load average was about 12 from concurrent worktrees. Nothing in this branch touches the realtime relay.
  • bun run structure:check: pass. bun run privacy:scan: pass. tests/ci-workflows/file-size-ratchet.test.ts, structure-ssot.test.ts and repo-hygiene.test.ts: 76/76.
  • I did not run a separate full bun run test; test:changed above already covered 1531 files.

Ratchet hot files touched (line counts after this branch)

File Lines Limit
src/server/index/serve-options.ts 1981 < 2000 (not in baseline)
scripts/test-layout/layout.json 1988 not ratcheted
src/server/index/websocket-handler.ts 444 < 2000
src/server/lifecycle.ts 616 < 2000
src/server/responses/core-opaque-recovery.ts 564 < 2000
src/adapters/openai-chat/passthrough.ts / reasoning-replay.ts 214 / 137 < 2000

Files not touched: passthrough-dispatch.ts, provider-routes.ts, openai-chat.ts, auth-context.ts, combos.test.ts.

Checklist

  • Scope stays focused and avoids unrelated cleanup. The only non-carried change is the WebSocket archive mapping from review.
  • Docs or release notes were updated when needed: ADR-7369, native-reasoning-retention.md, responses-failover.md, inbound-compat.md, byte-accounting.md, and docs-site server config from fix(responses): cap aggregate reasoning archive storage #7309.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults. The archive refusals expose no path or ciphertext on HTTP or WS, and opaque recovery cannot hide a resend on unrelated errors. No auth, credential or workflow files changed.

Co-authored-by: Ingwannu 186453546+Ingwannu@users.noreply.github.com
Co-authored-by: luvs01 27862058+luvs01@users.noreply.github.com
Co-authored-by: vadymhimself 11277453+vadymhimself@users.noreply.github.com

Ingwannu and others added 10 commits October 20, 2026 05:49
Preflight escaped-JSON size and reserve translator budget before materializing native Chat reasoning replay. Preserve existing budget errors, caller objects and retry/key-rotation ownership; retain synthetic regressions and update accounting contracts.
Distinguish the fixed serialized-request cap from the full final-size transient translator reservation. Document changed-replay ordering and finally release without changing production or test behavior.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…the retry's

The guard holds pre-content events across an empty-completion retry and
releases them with the retry's output. With nothing between the two attempts,
an unsigned reasoning delta from the retry is appended to the discarded
attempt's still-open reasoning item, so the client sees one reasoning item
whose text runs both attempts together ("discardedretry").

Each retry point now queues an assistant_boundary after the held events (only
when something is held), which the bridge and the direct encoders already
treat as "close the open items".

Co-Authored-By: Claude Code <noreply@anthropic.com>
A drain past its deadline calls abortAndReleaseAllTurns, which aborts every
controller bound to an admitted turn. trackStreamLifetime only stopped on a
consumer cancel, and several callers (the adapter bridge, run-turn execution,
client-encoder delivery, policy refusals, local SSE) pass a fresh
AbortController that nothing else listens to. For those turns the abort was a
no-op: the stream kept relaying upstream output and its source stayed open.

trackStreamLifetime now listens for the controller's abort and, once
tracking has started, finishes the turn, cancels the source and errors the
stream. The listener is attached after registerTurn so the lease's own abort
handling still runs first, and a controller registerTurn aborted
synchronously (a settled lease) keeps the previous behaviour.

Co-Authored-By: Claude Code <noreply@anthropic.com>
A throwing onDone callback must not leave the source open.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…transport

The HTTP boundary already maps typed archive failures to a sanitized 503 with a stable code. The WebSocket pre-response catch sent them as a raw 502 proxy_error frame that carried the archive path. Apply the same mapping there.
@lidge-jun
lidge-jun requested a review from Ingwannu as a code owner October 19, 2026 20:57
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-19T21:02:40.841420Z 88473ed PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Oct 19, 2026
@coderabbitai

coderabbitai Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration
  • Configuration used: Repository: lidge-jun/opencodex/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: d6d8083a-6c2a-4017-aca8-e7363e7a9f0f
📥 Commits

Reviewing files that changed from the base of the PR and between 79fa97b and 88473ed.

📒 Files selected for processing: 32
  • docs-site/src/content/docs/reference/configuration/server.md
  • scripts/test-layout/layout.json
  • src/adapters/openai-chat/passthrough.ts
  • src/adapters/openai-chat/reasoning-replay.ts
  • src/adapters/openai-responses/transport.ts
  • src/responses/reasoning-retention.ts
  • src/server/chat-native.ts
  • src/server/index/serve-options.ts
  • src/server/index/websocket-handler.ts
  • src/server/lifecycle.ts
  • src/server/responses/combo-stream-preflight.ts
  • src/server/responses/core-opaque-recovery.ts
  • src/server/responses/empty-completion-guard.ts
  • src/server/responses/reasoning-archive-error.ts
  • structure/data-planes/inbound-compat.md
  • structure/decisions/ADR-7369-cold-opaque-state-recovery.md
  • structure/providers-and-adapters.md
  • structure/transports/byte-accounting.md
  • structure/transports/native-reasoning-retention.md
  • structure/transports/responses-failover.md
  • structure/transports/responses.md
  • tests/fixtures/test-layout-expected.json
  • tests/responses/chat-native-reasoning-scope.test.ts
  • tests/responses/empty-completion-guard.test.ts
  • tests/responses/reasoning-archive-http.test.ts
  • tests/responses/reasoning-archive-ws.test.ts
  • tests/responses/reasoning-retention-quota.test.ts
  • tests/responses/reasoning-retention.test.ts
  • tests/responses/responses-azure-opaque-blob-recovery.test.ts
  • tests/responses/responses-cold-opaque-recovery.test.ts
  • tests/responses/responses-forward-transport-baseurl.test.ts
  • tests/service/shutdown-drain.test.ts
+----------------------------------------+
| If it was easy, it'd be called printf. |
+----------------------------------------+
  \
   \   (\__/)
       (•ㅅ•)
       /   づ
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 88473ed413

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +294 to 295
return parsed._stripReasoningEncryptedContent !== true
&& outboundResponsesBodyCarriesOpaqueBlob(outboundBody);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve keepalives during cold opaque preflight

When an ordinary same-provider turn carries encrypted_content, this broader predicate now sends it through the awaited preflight in passthrough-dispatch.ts, even without the prior retention marker. preflightComboStreamResponse buffers response.created, response.heartbeat, and SSE keepalive comments until semantic output, a terminal, or the large buffer cap, so a slow first token can leave the downstream connection completely idle long enough for a client or intermediary timeout despite the upstream remaining healthy. Please either limit the cold path to histories that actually need recovery or use a preflight wrapper that can forward liveness keepalives while retaining the ability to replace a proven pre-output rejection.

AGENTS.md reference: src/AGENTS.md:L17-L19

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants