Skip to content

feat(react): incrementally hydrate SSR state from streamed deltas - #4090

Open
ntucker wants to merge 30 commits into
masterfrom
cursor/nextjs-streamed-ssr-handoff-911d
Open

ntucker wants to merge 30 commits into
masterfrom
cursor/nextjs-streamed-ssr-handoff-911d

Conversation

@ntucker

@ntucker ntucker commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Motivation

A streaming HTML response is a sequence of committed revisions. A store serialized once — after a quiet window, or after useReadyCacheState() — cannot describe that sequence. Late islands arrive as markup whose endpoint/meta never landed in the browser, so useSuspense() takes the fetch branch on an empty seed.

PR 4090 already replaced the Next 10 ms createPersistedStoreServer snapshot on the wire: an inert baseline plus per-flush StateDeltas. That is not the remaining bug. Head still folds into the live store only from StreamedStateReceiver’s layout effect, has no per-key waiter, and lets getSnapshotStore() return from the empty baseline while RSC runs useLive(). Generic renderToPipeableStream / Anansi still emit one dataclient JSON blob.

Lane B (this tip): stop claiming zero-refetch / per-key waiters as shipped behavior. The Next 16 RSC-first race remains open and is out of Lane B scope. Do not ship Lane A (fold-on-script + per-key waiters + event-order tests) in this PR.

Secondary defects, same PR: hydration reads must overlay the server snapshot so a late boundary does not mismatch the live store (Gap B); </script> in API data must be escaped; Next managers must be a per-request factory.

Solution

This tip ships: the wire (inert baseline + StateDelta + HYDRATE), three-way merge, server-snapshot overlay, Next managers factory, and honest release surfaces. Per-key waiters, fold-on-script-arrival, and generic renderToPipeableStream baseline+delta are not in this release.

Wire format (@data-client/core). StateDelta, diffState / applyStateDelta / mergeStateDelta / selectBaseline / overlayState, and HYDRATE (hydrateReducer: ignore after client reset; reset opcode for server resets). Three-way merge: a slot the client changed keeps the client value. No React, Next, or Anansi imports here.

Next transport (@data-client/react/nextjs). useServerInsertedHTML() emits the baseline then deltas into the HTML stream. Live HYDRATE runs from StreamedStateReceiver’s layout effect. initialState is the one-time seed. Keep nonce, escaped JSON, and managers={() => Manager[]} (array throws on this entry).

Hydration reads. Browser useCacheState uses useSyncExternalStore with a server snapshot overlay so a boundary whose piece is late hydrates against what the server rendered, then converges to live state. React 19 is the DOM-preserving target; React 18 may client-render a still-dehydrated boundary when the store updates.

managers factory. Function form is the long-term API. Next entry: function only. Browser DataProvider: function supported, array transitional.

Docs honesty (Lane B). Changeset, blog, SSR guide, Next example README, the data-client-react SSR reference (no standalone data-client-ssr skill), and JSDoc describe what this tip does. They do not claim a per-key waiter or zero-refetch. The intended client clock (waiters + fold-on-script) is labeled as future work. The skill does not forbid a DOMContentLoaded interim. Figures: an overview with black boxes, then one zoom per box (docs/core/diagrams/_streamed_hydration.mdx). Labels are RSC and renderToPipeableStream.

The intended coordinator (fold-on-script, per-key useSuspense waiters, generic renderToPipeableStream baseline+delta, event-order tests) is Lane A / future client-clock work, not this tip.

Open questions

  • Waiters / fold-on-script remain future client-clock work. Per-key waiters, fold-on-script-arrival (independent of StreamedStateReceiver’s layout effect), and generic renderToPipeableStream / Anansi baseline+delta emission are intended, not shipped. Release surfaces now match this tip.
  • Next 16 RSC-first race remains open (consumer validation on 1dd5bfd). Out of Lane B scope. A document-wide DOMContentLoaded wait is an acceptable interim, not the required sole fix.
  • Next still has no API to emit a delta before the RSC payload that can start the Client Component. An upstream RSC-ordering or endpoint-manifest protocol is follow-up, not a substitute for waiters.
  • Absent vs not-yet-arrived keys are indistinguishable until stream close. That distinction is waiter work.
  • Server initManager disposers remain discarded (no request-end hook). Server managers stay inert.
  • Overlay has no tombstones: data the browser fetched during streaming can fill slots the server removed. Documented limitation.
  • No runtime deprecation warning yet for browser managers={array} (@data-client/test / Vue still construct arrays).
  • React 18 (stable and the canaries bundled with Next 13/14): data is correct, but a store update while a boundary is still dehydrated can make React client-render it. Documented as a limitation; React 19 is the fully supported target.
  • Nathaniel HOLD on squash still stands. Do not merge without an explicit go-ahead.
Open in Web Open in Cursor 

cursoragent and others added 9 commits September 11, 2026 15:48
Adds diffState/applyStateDelta/mergeStateDelta/selectBaseline under
state/stream plus a hydrateReducer that three-way merges streamed server
state into the live store, keeping slots the client changed and ignoring
deltas after a client reset. Exposed through __INTERNAL__ for the React
SSR adapters.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
…of its script tag

ServerData wrote JSON.stringify output directly into a <script>, so a
string containing </script> ended the tag early. Escape <, >, &, U+2028
and U+2029 as JSON unicode escapes, which keeps the payload valid JSON.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
createServerStore builds the redux-style store, controller and manager
wiring once so the Express and Next.js adapters stop duplicating it.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
…ive state

useCacheState now uses useSyncExternalStore's getServerSnapshot in the
browser so a Suspense boundary that hydrates after the store has already
changed (manager updates, streamed data) still matches the server HTML.
Updates keep flowing through StateContext, so transitions remain
non-blocking. DataProvider provides the snapshot through the internal
ServerSnapshotContext; React 16/17 and React Native keep the plain
context read.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
…et window

The Next.js DataProvider decided the server store was complete after a
fixed 10ms without in-flight fetches, so any async Server Component that
resolved later left #data-client-data empty and the client refetched
everything it had just received as HTML.

The provider now registers useServerInsertedHTML: the shell carries the
state so far as an inert JSON baseline and every later flush prepends a
delta script describing what changed since. The client folds these into
the hydration snapshot and merges them into the live store (HYDRATE) as
they arrive, so each Suspense boundary hydrates against exactly what it
was rendered from.

Also: nonce prop for CSP, managers accepted as a factory so each request
gets its own instances (arrays are ignored on the server with a warning),
and a local next/navigation declaration under typings/ that is not shipped.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
…gnored typings/ directory

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
…er streaming hydration in jsdom

A boundary whose data the server never sent (a delta lost to a
serialization failure, or arriving after its boundary hydrated) would
suspend on every hydration retry because the snapshot alone never gains
that data. The hydration view now overlays the server snapshot on the
live store, so such boundaries hydrate after exactly one client fetch
while everything the server did send still wins.

Adds jsdom coverage for both the plain DataProvider (late boundary
hydrates against server state, StrictMode, transitions, legacy branch)
and the Next.js client (deltas streaming into dehydrated boundaries,
queued deltas, client reset, missing baseline, document still loading,
managers factory). Test fixtures under __tests__/fixtures are no longer
collected as suites.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
…e and limitations

Also exports NextDataProviderProps, lists react-dom as an optional peer
for the nextjs entry, and makes the Next.js example exercise a Server
Component that resolves after the shell.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
…caping

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@changeset-bot

changeset-bot Bot commented Sep 11, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: ffed9ea

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 9 packages
Name Type
@data-client/react Minor
@data-client/img Minor
@data-client/test Minor
@data-client/core Minor
@data-client/vue Minor
example-benchmark-react Patch
test-bundlesize Patch
coinbase-lite Patch
example-benchmark Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Sep 11, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs-site Ignored Ignored Preview Sep 15, 2026 8:44pm UTC

Request Review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmark Spread

Details
Benchmark suite Current: ffed9ea Previous: 38dbb52 Ratio
setOneEntity in 10k entity store 151 ops/sec (±0.55%) 153 ops/sec (±0.97%) 1.01

This comment was automatically generated by workflow using github-action-benchmark.

Comment thread packages/react/src/server/__tests__/escapeJsonForHtml.node.tsx Fixed

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmark React

Details
Benchmark suite Current: ffed9ea Previous: 3fa47c8 Ratio
data-client: getlist-100 145.99 ops/s (± 5.5%) 137.94 ops/s (± 4.5%) 0.94
data-client: getlist-500 44.25 ops/s (± 5.8%) 44.05 ops/s (± 3.8%) 1.00
data-client: update-entity 416.67 ops/s (± 10.0%) 384.62 ops/s (± 9.3%) 0.92
data-client: update-user 408.33 ops/s (± 9.6%) 333.33 ops/s (± 8.6%) 0.82
data-client: getlist-500-sorted 50.51 ops/s (± 8.3%) 43.11 ops/s (± 9.4%) 0.85
data-client: update-entity-sorted 370.37 ops/s (± 9.7%) 312.5 ops/s (± 7.9%) 0.84
data-client: update-entity-multi-view 392.31 ops/s (± 8.7%) 327.96 ops/s (± 7.5%) 0.84
data-client: list-detail-switch-10 11.35 ops/s (± 7.7%) 9.9 ops/s (± 9.4%) 0.87
data-client: update-user-10000 81.63 ops/s (± 13.5%) 72.2 ops/s (± 14.8%) 0.88
data-client: invalidate-and-resolve 38.54 ops/s (± 6.2%) 37.67 ops/s (± 5.6%) 0.98
data-client: unshift-item 232.56 ops/s (± 5.2%) 219.81 ops/s (± 6.0%) 0.95
data-client: delete-item 312.5 ops/s (± 4.9%) 285.71 ops/s (± 3.8%) 0.91
data-client: move-item 188.68 ops/s (± 9.9%) 177.01 ops/s (± 7.9%) 0.94

This comment was automatically generated by workflow using github-action-benchmark.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmark

Details
Benchmark suite Current: ffed9ea Previous: 93555f8 Ratio
normalizeLong 448 ops/sec (±3.65%) 447 ops/sec (±4.75%) 1.00
normalizeLong Values 397 ops/sec (±1.10%) 408 ops/sec (±1.51%) 1.03
normalizeLong Scalar 355 ops/sec (±3.39%) 352 ops/sec (±3.69%) 0.99
normalizeLong Scalar update 857 ops/sec (±1.17%) 895 ops/sec (±0.68%) 1.04
denormalizeLong 229 ops/sec (±6.46%) 233 ops/sec (±6.00%) 1.02
denormalizeLong Values 221 ops/sec (±5.51%) 213 ops/sec (±4.94%) 0.96
denormalizeLong donotcache 1068 ops/sec (±1.00%) 1002 ops/sec (±0.64%) 0.94
denormalizeLong Values donotcache 759 ops/sec (±0.20%) 737 ops/sec (±0.59%) 0.97
denormalizeLong Scalar donotcache 1176 ops/sec (±0.26%) 1073 ops/sec (±0.13%) 0.91
denormalizeShort donotcache 500x 1352 ops/sec (±0.08%) 1437 ops/sec (±0.29%) 1.06
denormalizeShort 500x 590 ops/sec (±6.71%) 639 ops/sec (±6.97%) 1.08
denormalizeShort 500x withCache 7223 ops/sec (±1.02%) 6834 ops/sec (±5.52%) 0.95
queryShort 500x withCache 3363 ops/sec (±0.72%) 3206 ops/sec (±0.97%) 0.95
buildQueryKey All 51329 ops/sec (±1.52%) 58478 ops/sec (±1.39%) 1.14
query All withCache 5993 ops/sec (±2.78%) 5828 ops/sec (±2.46%) 0.97
denormalizeLong with mixin Entity 211 ops/sec (±8.08%) 209 ops/sec (±7.50%) 0.99
denormalizeLong withCache 7450 ops/sec (±0.22%) 7517 ops/sec (±0.32%) 1.01
denormalizeLong withCache (Scalar churn) 7368 ops/sec (±0.89%) 7491 ops/sec (±0.24%) 1.02
denormalizeLong Values withCache 6511 ops/sec (±1.19%) 5132 ops/sec (±1.60%) 0.79
denormalizeLong Scalar withCache 7374 ops/sec (±0.38%) 7648 ops/sec (±0.98%) 1.04
denormalizeLong Scalar update withCache 5510 ops/sec (±1.04%) 4074 ops/sec (±0.24%) 0.74
denormalizeLong All withCache 6130 ops/sec (±0.28%) 6058 ops/sec (±0.18%) 0.99
denormalizeLong Query-sorted withCache 6234 ops/sec (±2.10%) 6098 ops/sec (±1.48%) 0.98
denormalizeLongAndShort withEntityCacheOnly 1581 ops/sec (±1.64%) 1748 ops/sec (±0.19%) 1.11
denormalize bidirectional 50 4138 ops/sec (±10.27%) 4498 ops/sec (±10.41%) 1.09
denormalize bidirectional 50 donotcache 44486 ops/sec (±0.22%) 42385 ops/sec (±1.43%) 0.95
getResponse 5075 ops/sec (±4.54%) 4418 ops/sec (±4.05%) 0.87
getResponse (null) 10275618 ops/sec (±0.41%) 10236651 ops/sec (±0.70%) 1.00
getResponse (clear cache) 204 ops/sec (±9.23%) 203 ops/sec (±7.07%) 1.00
getSmallResponse 3810 ops/sec (±1.20%) 3543 ops/sec (±0.24%) 0.93
getSmallInferredResponse 3023 ops/sec (±0.32%) 2852 ops/sec (±1.79%) 0.94
getResponse Collection 4661 ops/sec (±5.99%) 4306 ops/sec (±4.05%) 0.92
get Collection 2826 ops/sec (±0.32%) 2707 ops/sec (±0.19%) 0.96
get Query-sorted 5889 ops/sec (±2.54%) 5052 ops/sec (±1.47%) 0.86
setLong 448 ops/sec (±0.27%) 467 ops/sec (±0.60%) 1.04
setLongWithMerge 257 ops/sec (±0.23%) 257 ops/sec (±0.45%) 1
setLongWithSimpleMerge 268 ops/sec (±0.42%) 272 ops/sec (±0.80%) 1.01
setSmallResponse 500x 873 ops/sec (±1.42%) 926 ops/sec (±1.48%) 1.06

This comment was automatically generated by workflow using github-action-benchmark.

cursoragent and others added 3 commits September 11, 2026 16:48
Co-authored-by: Nathaniel Tucker <me@ntucker.me>
Manager instances cannot be shared between server requests, so the array
form only ever applied to the browser while the server kept its defaults.
The prop now takes a function, called once per request on the server and
once in the browser; an array throws with a migration hint.

BREAKING CHANGE: @data-client/react/nextjs DataProvider managers prop
changes from Manager[] to () => Manager[].

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
…kfile

The nextjs managers change ships as a minor, so @data-client/img and
@data-client/test must accept 0.19.x; also records the optional react-dom
peer in yarn.lock.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@ntucker

ntucker commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

CHANGE_THIS_PR

The streaming handoff design is the right shape for this problem — baseline + deltas via useServerInsertedHTML, three-way HYDRATE merge, and useSyncExternalStore overlay for Gap B. I’m not asking to unwind that. TanStack-style dehydrate/HydrationBoundary doesn’t replace a normalized entity graph with in-place GC; this isn’t idle complexity.

What blocks Ready (adversarial pass kept these):

  1. yarn.lockreact-dom was added as an optional peer on @data-client/react but the lockfile wasn’t updated. Vercel deploy fails with YN0028 (immutable install would modify the lockfile). Commit the lockfile refresh.

  2. React 17 matrix — new hydration.web.tsx / provider-hydration.web.tsx statically import react-dom/client, so the suite never loads. The existing LegacyReact ? describe.skip never runs. Lazy/require the client APIs (or otherwise keep the import off the React 17 path) so the skip can work.

  3. React 18 matrixunit_tests-^18 is red while latest is green. Docs already scope DOM-preserving hydration to React 19 (flushSync during a dehydrated boundary can client-render on 18). Gate __server / “no recoverable warning” asserts to React 19 — do not invent a DOM-preserving React 18 receive path. Separately, reset-vs-delta and late Probe/liveTitle convergence still need an 18-aware fix or precise asserts so 18 covers live-store guarantees without over-claiming React 19 behavior.

  4. ActionTypesHydrateAction — prefer shipping @data-client/core as a minor, not patch. Exhaustive manager switches break either way; patch makes the bump harder to justify after release. Flip the changeset before Ready.

Not asking in this PR: managers factory (array was already unused for the server store; warn + factory is an improvement), soft-nav / router.refresh streaming, Express 10ms path, CodeQL on the double-stringify escape path (looks like a false positive), or a redesign of overlay/merge/flushSync.

Draft is fine; Ready needs green ^17/^18, the lockfile, and the semver flip.

@github-actions

github-actions Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Size Change: +778 B (+0.96%)

Total Size: 81.8 kB

📦 View Changed
Filename Size Change
examples/test-bundlesize/dist/rdcClient.js 11.6 kB +778 B (+7.17%) 🔍
ℹ️ View Unchanged
Filename Size
examples/test-bundlesize/dist/App.js 1.46 kB
examples/test-bundlesize/dist/polyfill.js 307 B
examples/test-bundlesize/dist/rdcEndpoint.js 8.07 kB
examples/test-bundlesize/dist/react.js 59.6 kB
examples/test-bundlesize/dist/webpack-runtime.js 784 B

compressed-size-action

- a delta that ends the baseline wait before DOMContentLoaded no longer
  lets the stale load listener clear the receiver installed later; the
  pending wait is cleared once settled
- the client managers factory result is cached on the per-document
  snapshot store so StrictMode's double useMemo cannot run it twice
- Controller is honored on the server store and SSR controller; gcPolicy
  is documented as browser-only; store providers no longer receive props
  consumed at store creation
- only StateDelta and StateBaseline are public; createServerStore drops
  its unused managers return

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@ntucker
ntucker marked this pull request as ready for review September 11, 2026 17:27
@ntucker

ntucker commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

CHANGE_THIS_PR (updated for 284b306)

Prior note raced this push. Cleared on this head:

  1. yarn.lock / optional react-dom peer — lockfile includes it; good.
  2. ActionTypesHydrateAction as core minor — changeset flipped; good.
  3. Managers factory as a hard break (array throws + migration hint) — fine; keep it.

Still blocks Ready (adversarial pass kept these; CircleCI unit_tests-^17 / ^18 still red on this SHA):

  1. React 17 matrixhydration.web.tsx / provider-hydration.web.tsx still statically import react-dom/client, so the suite never loads (Cannot find module at import). LegacyReact ? describe.skip never runs. Lazy/require the client APIs after the version gate (or only inside describeHydration) so the skip works. Do not exclude the whole provider-hydration file — describe('useCacheState branches') is outside describeHydration and is the React 16/17 shape coverage.

  2. React 18 matrix — docs already scope DOM-preserving hydration to React 19. Gate __server / empty-errors (no recoverable client-render warning) to React 19 — do not invent a DOM-preserving React 18 receive path. On ^18, streamsIntoDehydratedBoundary already gets liveTitle right and only dies on __server; keep those live-store asserts. For reset-vs-delta liveTitle and mid-hydration Probe text: prefer asserting via store.getState() / controller state, or gate Probe DOM expectations to R19 — overlay/getServerSnapshot can win during hydration, so those failures are not a license to “fix” a phantom live receive path. Do not skip the whole describeHydration on ^18 (folds-deltas, baseline/loading, managers-factory still pass).

Not asking: soft-nav streaming, Express 10ms path, CodeQL on the double-stringify / test regexes (false positives), website GHA HomepageFeatures SvgComponent typecheck (unrelated; CircleCI typecheck green), or redesign of overlay/merge/flushSync.

Draft is fine; Ready needs green ^17/^18.

@ntucker

ntucker commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

CHANGE_THIS_PR (updated for 7dd9a6f)

7dd9a6f cleared the non-matrix review items: Controller honored on server/client, managers factory cached for StrictMode, DOMContentLoaded race fixed, public stream types trimmed. Good — keep those.

Still blocks merge (adversarial pass kept these; CircleCI unit_tests-^17 / ^18 still red on this SHA). PR is Ready while those are red — fine as process note; the matrix fix is the unblocker.

  1. React 17 matrixhydration.web.tsx / provider-hydration.web.tsx still statically import react-dom/client, so the suites never load (Cannot find module at import). LegacyReact ? describe.skip never runs. Lazy/require the client APIs after the version gate (or only inside describeHydration) so the skip works. Do not exclude the whole provider-hydration file — describe('useCacheState branches') is outside describeHydration and is the React 16/17 shape coverage.

  2. React 18 matrix — docs already scope DOM-preserving hydration to React 19. Gate __server / empty-errors (no recoverable client-render warning, including the transition case) to React 19 — do not invent a DOM-preserving React 18 receive path. On ^18, streamsIntoDehydratedBoundary already gets liveTitle/text right and only dies on __server; keep those live-store asserts. For reset-vs-delta liveTitle and mid-hydration Probe text: prefer asserting via store.getState() / controller state (Probe/useCache can still see overlay/getServerSnapshot during hydration; that is not a missing live receive path). Gating Probe DOM expectations to R19 is an acceptable backup. Do not skip the whole describeHydration on ^18 (folds-deltas, baseline/loading, DOMContentLoaded race, managers-factory, Controller still pass).

Not asking: soft-nav streaming, Express 10ms path, CodeQL / GHA website typecheck (unrelated; CircleCI typecheck green), or redesign of overlay/merge/flushSync.

Ready needs green ^17/^18.

cursoragent and others added 2 commits September 11, 2026 17:36
Lazy-require react-dom/client after the LegacyReact gate so describe.skip
can run. Gate DOM-preserving / no-recoverable-warning asserts to React 19
and check reset-vs-delta and mid-hydration Probe via the live store.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
On 18, setResponse while a sibling is dehydrated is deferred, and a
client reset can remount from the snapshot. Gate those live-store
claims to React 19; keep data, refetch, and stream liveTitle asserts.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@codecov

codecov Bot commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.78238% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.59%. Comparing base (1ccc7ab) to head (ffed9ea).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
packages/core/src/state/stream/writeDelta.ts 90.00% 1 Missing and 5 partials ⚠️
packages/core/src/state/stream/diffState.ts 92.30% 0 Missing and 3 partials ⚠️
packages/core/src/state/stream/selectBaseline.ts 86.95% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4090      +/-   ##
==========================================
- Coverage   97.84%   97.59%   -0.25%     
==========================================
  Files         156      167      +11     
  Lines        3057     3243     +186     
  Branches      612      661      +49     
==========================================
+ Hits         2991     3165     +174     
- Misses         18       21       +3     
- Partials       48       57       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Drop the unused-generality id helper; R19 probe covers the late-update
store read, and reset asserts getState() inline.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@ntucker

ntucker commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

Staff LGTM (updated for f123190)

Prior matrix CHANGE_THIS_PR items are cleared on this head:

  1. React 17react-dom/client is jest.requireActual after the LegacyReact gate in both hydration suites; describe('useCacheState branches') still runs outside describeHydration. CircleCI unit_tests-^17 green.
  2. React 18__server / empty-errors (transition + late-delta) gated to React 19; mid-hydration store writes and reset-vs-delta live-store drop asserted on 19 only (18 defers updates / may remount — documented). Stream liveTitle, folds-deltas, refetch counts, and the rest of describeHydration still run on ^18. CircleCI unit_tests-^18 green on 7aeb7fa and f123190.

Adversarial pass on the R18 re-spec: KILL as CHANGE_THIS_PR — honest React 18 hydration scheduling, not a missing product path. Optional FOLLOW_UP only if we ever need stronger R18 guarantees around reset-during-streaming vs remount/snapshot fold — not this PR.

Hold squash for Bugbot success. GHA “Typecheck website” / CodeQL remain the prior unrelated / false-positive set; CircleCI typecheck is green.

No further changes requested on this PR.

@ntucker

ntucker commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

Staff review — LGTM / FOLLOW_UP only @ f123190

CHANGE_THIS_PR items from 7dd9a6f are cleared on this head. No new CHANGE_THIS_PR.

Cleared

  • React 17: hydrateRoot via jest.requireActual after LegacyReact gate in hydration.web.tsx / provider-hydration.web.tsx; useCacheState branches kept; unit_tests-^17 PASS
  • React 18: __server / empty-errors gated to R19; describeHydration still runs; live-store coverage kept; unit_tests-^18 PASS
  • Prior clears stand: lockfile optional react-dom, core HYDRATE minor, Controller prop, managers factory StrictMode cache, DOMContentLoaded race, stream API trim

Architecture still LGTM: baseline+deltas, three-way HYDRATE, useSyncExternalStore overlay — right shape for normalized SSR handoff.

FOLLOW_UP (not blocking)

  • GHA typecheck / CodeQL red look like the previously noted unrelated website / FP class (CircleCI typecheck green) — confirm before squash if anything new
  • Holding squash for Bugbot SUCCESS (still pending on this HEAD)

Ready for merge once Bugbot is green (or waived).

@cursor
cursor Bot force-pushed the cursor/nextjs-streamed-ssr-handoff-911d branch from d7ecc8c to f123190 Compare September 11, 2026 18:30
@ntucker

ntucker commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

Staff LGTM (updated after second force-push back to f123190)

head_ref_force_pushed at 18:30 UTC dropped the re-landed browser DataProvider managers factory (d7ecc8c / 114d7e5). Current tip is again f123190.

CHANGE_THIS_PR cleared

  • Prior blocks on that tip were: (1) StrictMode once-call assert (unit_tests-^17 / ^18 Expected 1, Received 2), and (2) stale “browser unchanged / Manager[] only” sentence in .changeset/nextjs-managers-factory.md contradicting the factory docs. Reverting removes both without a document-level managers cache — correct call for this PR.
  • Do not re-land the browser factory inside this branch unless the test is re-spec’d (stable across re-renders within a mount; version-aware / drop StrictMode once-claim) and the Next.js managers changeset no longer claims the browser is unchanged.

Prior LGTM at f123190 still stands

  • Streaming/hydration design unchanged: baseline + deltas, three-way HYDRATE, useSyncExternalStore overlay.
  • Matrix gates, lockfile, core HYDRATE minor, Next.js managers factory, Controller, DOMContentLoaded race — all still good.
  • Browser DataProvider remains managers?: Manager[] on this tip; .changeset/nextjs-managers-factory.md “browser unchanged” line is accurate again.

Hold squash for

  • CircleCI re-run after this force-push (same tip was green before at ~18:09)
  • Bugbot SUCCESS (or waive) — in progress on this HEAD

Optional FOLLOW_UP only (not this PR): browser DataProvider managers as Manager[] | (() => Manager[]) with the honest StrictMode call-count claim + matching changeset wording.

No further changes requested on this PR.

@ntucker

ntucker commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator Author

CHANGE_THIS_PR (updated for d7ecc8c)

Prior Staff LGTM at f123190 still covers the streaming/hydration design. Tip is again d7ecc8c (browser DataProvider managers factory + docs). Shape is fine (array still works; resolve once per mount via useRef; multi-provider isolation is the right product claim). Adversarial pass kept the items below.

Blocks Ready

  1. React 17 / 18 matrix — CircleCI unit_tests-^17 / unit_tests-^18 red on this SHA. Sole failure: provider.tsxmanagers factory › is called once, even across re-renders and under StrictMode — Expected 1, Received 2. unit_tests-latest (React 19) stays green.

    Once-per-mount via useRef matches Controller / gcPolicy. Under Strict Mode the render function (and each useRef) runs twice and one pass is discarded — that purity check calls the factory on the discarded pass; it is not a second production mount, and initManager effects never run on the discarded instance. Next.js snapshotStore.managers caching is a different lifetime; do not copy it onto browser DataProvider just to force call-count 1 (would fight multi-provider isolation and reuse cleaned-up instances).

    Fix (test-only): split the claim — assert the factory is stable across re-renders within a mount; drop or version-gate the “once under StrictMode” call-count (React 19 currently sees 1, 17/18 see 2). Keep init/cleanup pairing. Unconditional expect(2) would break latest.

  2. Stale Next.js managers changeset.changeset/nextjs-managers-factory.md still says the browser DataProvider is unchanged and still takes Manager[]. Sibling .changeset/dataprovider-managers-factory.md and the code now say arrays are transitional and the factory is supported. Fix or remove that one sentence so the release notes don’t contradict.

Not asking: GHA website typecheck / CodeQL (same HomepageFeatures SvgComponent / FP class as before; CircleCI typecheck green), soft-nav streaming, Express 10ms path, or redesign of overlay/merge/flushSync.

Ready needs green ^17/^18 again (and the changeset line).

cursoragent and others added 2 commits September 13, 2026 15:07
Once-per-mount via useRef is correct. StrictMode 17/18 discards a
render/useRef pass so the factory runs twice; React 19 keeps one.
Assert stability across re-renders within a mount and keep
init/cleanup pairing on the live instance.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
The browser DataProvider now accepts a factory; arrays are
transitional. Drop the stale "unchanged / Manager[] only" line so
release notes do not contradict the sibling changeset.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@ntucker

ntucker commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator Author

Staff LGTM (updated for 47e757f)

Prior CHANGE_THIS_PR at d7ecc8c is cleared on this tip:

  1. StrictMode once-call assert52535fc re-specs the managers factory test: version-aware mount call count (1 on React 19, 2 on 17/18), then asserts stability across re-renders within a mount and init/cleanup pairing on the live instance. No document-level managers cache — correct.
  2. Stale Next.js managers changeset47e757f drops the “browser unchanged / Manager[] only” line; now matches sibling dataprovider-managers-factory and the code.

Prior LGTM still stands for the streaming/hydration design (baseline + deltas, three-way HYDRATE, useSyncExternalStore overlay) and the earlier matrix / lockfile / core minor clears.

Hold squash for

  • CircleCI unit_tests-^17 / ^18 / latest green on this HEAD
  • Bugbot SUCCESS (or waive) — prior Bugbot note on d7ecc8c about Expected-1 was the old test claim; obsolete once this tip’s re-spec lands
  • Nathaniel’s earlier Hold still applies — do not merge without an explicit go-ahead

No further changes requested on this PR. Optional FOLLOW_UP only if we later want stronger R18 remount-from-snapshot guarantees — not this PR.

Rename escapeJsonForHtml to escapeJsonForScript: CodeQL treats any callee
named like an HTML sanitizer as producing output unsafe for code
construction, hiding the actual \u003c/U+2028 escaping behind the name.
Test regexes that parse generated <script> tags are now case-insensitive
and tolerate whitespace before the closing bracket.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@ntucker

ntucker commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator Author

Staff LGTM (updated for 1dd5bfd)

Prior CHANGE_THIS_PR at d7ecc8c cleared on 47e757f still stands. This tip is CodeQL hygiene only:

  1. Improper code sanitization FPescapeJsonForHtmlescapeJsonForScript (call sites + file rename). Escape set unchanged (\u003c/\u003e/\u0026/U+2028/U+2029). Name matches the real sink (inert JSON <script> and the double-stringify JSON.parse(...) literal in deltaScript).
  2. Bad HTML filtering regexp (tests) — case-insensitive + optional whitespace before > on test parsers only (escapeJsonForScript.node.tsx, streaming.node.tsx). Production escape path untouched.

No CHANGE_THIS_PR. Architecture LGTM unchanged: baseline + deltas, three-way HYDRATE, useSyncExternalStore overlay, managers factory with honest StrictMode claim.

Hold squash for

  • CircleCI unit_tests-^17 / ^18 / latest green on this HEAD
  • Bugbot SUCCESS — already green on 1dd5bfd
  • Nathaniel’s earlier Hold — do not merge without an explicit go-ahead

GHA typecheck (website HomepageFeatures SvgComponent) is the same prior unrelated failure; CircleCI typecheck has been the source of truth. CodeQL Analyze still pending on this SHA — expect the rename to clear the prior FP class.

No further changes requested on this PR. Optional FOLLOW_UP only if we later want stronger R18 remount-from-snapshot guarantees — not this PR.

@ntucker

ntucker commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator Author

CHANGE_THIS_PR — real Next 16 consumer validation found a client handoff race on 1dd5bfd.

I tested packed artifacts from this exact head in the order-book app that originally reproduced the 10 ms bug:

  • Next 16.3.4 / React 19.2.8
  • root-layout @data-client/react/nextjs provider
  • async route params + six useLive() consumers
  • custom WebSocket manager supplied through the new factory API

Server side passes: production build is clean; 8 concurrent mixed-symbol requests each returned 87 data cells and a route-isolated cache. The baseline is the 101-byte initial state and 3–5 streamed deltas reconstruct the expected OrderBook/ticker/trades/candles state.

Client side does not meet the zero-refetch claim on this head. On a clean hard navigation, browser PerformanceResourceTiming showed the SSR endpoints starting again immediately:

  • watchlist ticker: ~39 ms
  • symbol info: ~40 ms
  • order book / trades / candles: ~236 ms

(The later direct depth request from this app's Binance sequence synchronizer is expected and separate.) No hydration warning was required to trigger this; useful SSR HTML remained visible, but the Data Client requests were duplicated.

The raw response ordering is correct: each delta script precedes its dependent Fizz HTML. The failing assumption is that this also precedes Client Component execution. In Next 16, Flight can start that client work before the inserted HTML delta script executes and before StreamedStateReceiver commits its layout effect. Because the inert baseline is already present in <head>, getSnapshotStore() does not suspend, initializes from the empty baseline, and useSuspense() starts client fetches before later deltas are usable.

I validated the conservative fix below:

 const element = document.getElementById(BASELINE_ID);
-if (!element && document.readyState === 'loading') {
+if (document.readyState === 'loading') {
   throw (queue.pending ??= new Promise<void>(resolve => {
-    // whichever fires first wins; the loser must not clobber the receiver
     const done = () => {
       document.removeEventListener('DOMContentLoaded', done);
-      if (queue.onDelta === done) queue.onDelta = undefined;
       queue.pending = undefined;
       resolve();
     };
     document.addEventListener('DOMContentLoaded', done);
-    queue.onDelta = done;
   }));
 }

In other words: while the initial document is still parsing, wait through all streamed deltas; do not let the first delta end bootstrap. At DOMContentLoaded, fold the complete queue and initialize the client store once.

With only that runtime refinement:

  • no Data Client SSR endpoint refetched on startup;
  • the only browser REST request was the expected app-owned depth resync at 1839 ms;
  • DOMContentLoaded was 1268 ms (response start 82 ms), so the tradeoff is explicit: SSR pixels still stream, but Data Client interactivity waits for the full initial document;
  • 8 concurrent mixed routes still passed cache reconstruction/isolation;
  • app production build, TypeScript, and ESLint passed;
  • both hydration suites passed (15 tests);
  • all React package suites passed: 55 suites, 498 passed / 1 skipped, 127 snapshots.

The regression test needs the baseline installed before getSnapshotStore() with document.readyState = 'loading', then stream a delta and assert the thrown bootstrap promise is still unsettled until DOMContentLoaded. The existing test installs the baseline after the first call, so it does not cover the real condition. It should also require the thrown value to be a Promise rather than using optional chaining, and restore readyState in finally.

If preserving progressive interactivity is required, the alternative is a larger endpoint-aware client wait: missing useSuspense() keys must wait for the next server delta while the document is loading, retry after each delta, and only fetch client-side after DOMContentLoaded confirms no server value arrived. Raw HTML script order alone is not a sufficient synchronization contract on Next 16.

I would not merge the current head with the zero-client-request claim unchanged. The DOMContentLoaded gate is small and validated; a more progressive design needs an equivalent real Next 16 browser test that asserts request counts, not only final DOM/cache state.

Rewrite the SSR guide, changeset, blog, Next example README, and agent
skills around baseline-plus-delta hydration. Inline the sequence once via
the existing diagrams MDX extract. Keep Open questions honest: waiters
and fold-on-script are the remaining client clock, not this commit.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@cursor cursor Bot changed the title fix(react): stream Next.js SSR state per flush and hydrate cache reads from the server snapshot feat(react): incrementally hydrate SSR state from streamed deltas Sep 15, 2026
Rewrite the streaming test HTML parser so CodeQL does not treat a
</script> regex as a bad tag filter. Type homepage SVGs from the SVGR
import to avoid dual @types/react conflict in website typecheck.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@ntucker

ntucker commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

CHANGE_THIS_PR (adversarial re-check @ 49956a0)

Tip 49956a0 is CodeQL/SVG hygiene only. Runtime handoff is still the Sep 14 Flight-first race.

Verified on this SHA:

  • getSnapshotStore() returns as soon as the G0 baseline element is present (or uses initialState); it does not wait for per-key deltas. Comment admits intended fold-on-script / waiters; “Today this still suspends only when the baseline element is missing.”
  • StreamedStateReceiver is the only live HYDRATE path (layout effect + flushSync). Comment claims “the per-key waiter covers that race” — no waiter exists.
  • createPersistedStoreClient seeds DataProvider from that one-shot snapshot; later pieces are layout-effect folds only.
  • useSuspense has no stream-open waiter branch — miss → controller.fetch().
  • deltaQueue has baseline pending only; no per-key waiters.
  • Server-snapshot overlay fixes Gap B mismatch, not empty-G0 miss → refetch.

Release surfaces still overclaim the unfinished contract (changeset, docs/core/guides/ssr.md streamed section, data-client-ssr skill, v0.19 blog). PR Open questions honesty does not make that FOLLOW_UP while those ship as product behavior.

Before Ready — pick one:

  • (A) Land waiters + fold-on-script-arrival (independent of the receiver effect) + Flight-first tests that assert fetch counts / event order.
  • (B) Rewrite release surfaces to match tip: wire G0+deltas+HYDRATE may ship; per-key waiters / zero-refetch / generic Fizz baseline+delta are not shipped yet.

Do not treat DOMContentLoaded-always as the required sole fix (validated progressive tradeoff elsewhere; not the intended contract). Soft LGTM of hygiene while changelog/guide claim zero-refetch is not acceptable for a minor.

Architecture already on tip (baseline+deltas, three-way HYDRATE, overlay, Next managers factory) remains LGTM. No competing cloud agent from Staff — Coding owns any fix spawn.

Lane B honesty pass: changeset, blog, SSR guide, Next example README,
skill, and JSDoc now match the tip. Wire (G0 + StateDelta + HYDRATE)
ships; per-key waiters, zero-refetch, and generic Fizz incremental
emission do not. A DOMContentLoaded interim is not forbidden.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@ntucker

ntucker commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Staff LGTM (Lane B @ ec11b14)

CHANGE_THIS_PR at 1a686d7 / 49956a0 cleared on this tip: claim surfaces match Open questions.

  • Changeset / blog / guide / skill / JSDoc: wire format (G0 + StateDelta + layout-effect HYDRATE) shipped; per-key waiters / fold-on-script / Flight-first zero-refetch not in this release
  • Skill no longer forbids DOMContentLoaded; marks it an acceptable interim
  • Generic Fizz / Anansi stay one-shot; diagram marks the sequence as intended

Runtime race remains known/documented — Lane A is follow-up after this lands, not a blocker for Lane B honesty.

Bugbot SUCCESS on this HEAD. Nathaniel HOLD on squash stands — no merge from Staff.

Split the SSR hydration sequence into an overview with black boxes
and one zoom per box. Replace Fizz/Flight jargon with
renderToPipeableStream and RSC in the guide, skill, changeset, blog,
example README, and Next adapter comments.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@ntucker

ntucker commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Staff LGTM (updated for a72a3b5)

Lane B LGTM at ec11b14 still stands. This tip is docs/comment polish only:

  1. Diagrams: overview with black boxes + one zoom per box (replaces the long sequence)
  2. Product language: Flight → RSC, Fizz → renderToPipeableStream; G0 naming dropped from prose in favor of baseline
  3. Adapter JSDoc/comments only — no runtime behavior change
  4. Claim surfaces still honest: waiters / fold-on-script not shipped; RSC-first miss fetches; generic /ssr one-shot

Non-blocking nit: _streamed_hydration.mdx closing line still says “The sequence is the intended client clock” after the sequence was removed — prefer “figures” to match the guide. Not a Ready block.

Bugbot SUCCESS on this HEAD. Nathaniel HOLD on squash stands — no merge from Staff. Lane A (waiters + fold-on-script + RSC-first tests) remains follow-up after merge.

No CHANGE_THIS_PR. Coding: no spawn for this tip.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
SSR is a reference inside data-client-react, not a standalone skill.
Decision trees include an SSR column only where behavior differs; otherwise
they are the same as client (streamed baseline+deltas instead of one
initialState).

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@ntucker

ntucker commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Staff LGTM (updated for 960349c)

Lane B LGTM at ec11b14 / a72a3b5 still stands. This tip is skill catalog hygiene only — no runtime change.

  1. Fold — deletes standalone data-client-ssr; SSR/streaming lives in data-client-react (matches the product claim: same hooks, not a second data API).
  2. Pointers — manager + setup + agent-skills.md point at data-client-react; no leftover data-client-ssr skill tree on this tip.
  3. Claim honesty — waiters / fold-on-script / RSC-first zero-refetch still marked not shipped; DOMContentLoaded interim language preserved; Flight/Fizz stay out of prose.

Lane A (waiters + fold-on-script) remains follow-up after merge. HOLD squash until Nathaniel clears merge.

Revert the previous fold (per-row "Same" tables and intended-behavior claims
in SKILL.md). SSR now lives in references/ssr.md inside data-client-react and
documents only what differs from the browser: entry per host, streamed
baseline + StateDelta on Next.js App Router vs one initialState elsewhere, and
what is not shipped (per-key waiters, fold-on-script, streamed generic
renderToPipeableStream). No standalone data-client-ssr skill.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
@ntucker

ntucker commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Staff LGTM (updated for 53e107e)

Lane B LGTM at ec11b14 / a72a3b5 / 960349c still stands. This tip redoes the SSR skill fold — no runtime change.

  1. Reference — SSR lives in data-client-react/references/ssr.md: which entry, App Router stream, one-shot Express/renderToPipeableStream and Pages, and an explicit Not shipped section.
  2. SKILL.md — drops the per-row “Same” SSR columns; one line that hooks are identical and only seeding differs, with a pointer to the reference.
  3. Catalog — no standalone data-client-ssr; manager points at references/ssr.md; setup no longer duplicates the SSR blurb.
  4. Claim honesty — per-key waiters / fold-on-script / streamed generic renderToPipeableStream still not shipped; RSC-first miss fetches; DOMContentLoaded interim is acceptable, not the contract.

No CHANGE_THIS_PR. Architecture LGTM unchanged (baseline + deltas, three-way HYDRATE, overlay, Next managers factory).

Hold squash for

  • CircleCI / Bugbot green on this HEAD (checks still settling)
  • Nathaniel’s HOLD — do not merge without an explicit go-ahead

Lane A (waiters + fold-on-script + RSC-first fetch-count tests) remains follow-up after merge. Coding: no spawn for this tip.

cursoragent and others added 2 commits September 15, 2026 20:37
Add RSC as a trigger term in the data-client-react description, drop the
docs-terminology rule from references/ssr.md (not a host difference), and
state the SUBSCRIBE caveat without restating browser behavior.

Co-authored-by: Nathaniel Tucker <me@ntucker.me>
Co-authored-by: Nathaniel Tucker <me@ntucker.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants