Skip to content

[core] Prune schema modules from workflow bundles - #3550

Merged
NathanColosimo merged 7 commits into
mainfrom
codex/replay-prune-runtime-graph
Aug 25, 2026
Merged

[core] Prune schema modules from workflow bundles#3550
NathanColosimo merged 7 commits into
mainfrom
codex/replay-prune-runtime-graph

Conversation

@NathanColosimo

@NathanColosimo NathanColosimo commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • move runtime-only event metadata and attribute validation onto schema-free @workflow/world subpaths
  • keep the public Zod schemas available to hosts without pulling them into the sandbox
  • intentionally remove the superseded root-level World validation and event lookup exports as a v5 breaking change
  • add regression coverage for both a minimal workflow and normal attribute, hook, and step APIs

Result

For the 97-step benchmark workflow:

  • VM bundle: 612,980 B → 26,831 B (-95.6%)
  • fresh-context evaluation p50: 10.536 ms → 0.121 ms (-98.9%)
  • Zod modules in the benchmark VM bundle: 77 → 0 (workflows that import Zod themselves still include it)

Review scope

Serialized formats are unchanged. Besides the dependency-boundary refactor, this tightens two edge cases: an unknown attribute deletion can no longer offset a newly added key, and prototype-like event type strings safely return no metadata. The @workflow/world changeset is marked major for the intentionally removed public exports; on the current prerelease branch this still produces the next v5 beta.

Validation

  • full workspace typecheck
  • builders, core, world, and world-vercel package builds/typechecks
  • 3,233 package tests across builders, core, world, and world-vercel
  • focused workflow-bundle boundary regression tests
  • Next/Turbopack and Vite production builds
  • clean Codex + Claude autoreview after addressing feedback

@changeset-bot

changeset-bot Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 4834383

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 20 packages
Name Type
@workflow/builders Patch
@workflow/core Patch
@workflow/world Major
@workflow/astro Patch
@workflow/cli Patch
@workflow/nest Patch
@workflow/next Patch
@workflow/nitro Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch
@workflow/vitest Patch
@workflow/web-shared Patch
@workflow/web Patch
workflow Patch
@workflow/world-testing Patch
@workflow/world-local Patch
@workflow/world-postgres Patch
@workflow/world-vercel Patch
@workflow/nuxt Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
example-nextjs-workflow-turbopack Ready Ready Preview, v0 Aug 25, 2026 2:59am
example-nextjs-workflow-webpack Ready Ready Preview, v0 Aug 25, 2026 2:59am
example-workflow Ready Ready Preview, v0 Aug 25, 2026 2:59am
workbench-astro-workflow Ready Ready Preview, v0 Aug 25, 2026 2:59am
workbench-express-workflow Ready Ready Preview, v0 Aug 25, 2026 2:59am
workbench-fastify-workflow Ready Ready Preview, v0 Aug 25, 2026 2:59am
workbench-hono-workflow Ready Ready Preview, v0 Aug 25, 2026 2:59am
workbench-nestjs-workflow Ready Ready Preview, v0 Aug 25, 2026 2:59am
workbench-nitro-workflow Ready Ready Preview, v0 Aug 25, 2026 2:59am
workbench-nuxt-workflow Ready Ready Preview, v0 Aug 25, 2026 2:59am
workbench-python-workflow Ready Ready Preview, v0 Aug 25, 2026 2:59am
workbench-sveltekit-workflow Ready Ready Preview, v0 Aug 25, 2026 2:59am
workbench-tanstack-start-workflow Ready Ready Preview, v0 Aug 25, 2026 2:59am
workbench-vite-workflow Ready Ready Preview, v0 Aug 25, 2026 2:59am
workflow-docs Ready Ready Preview, v0 Aug 25, 2026 2:59am
workflow-swc-playground Ready Ready Preview, v0 Aug 25, 2026 2:59am
workflow-tarballs Ready Ready Preview, v0 Aug 25, 2026 2:59am
workflow-web Ready Ready Preview, v0 Aug 25, 2026 2:59am

@NathanColosimo NathanColosimo mentioned this pull request Aug 14, 2026
Open
@NathanColosimo NathanColosimo mentioned this pull request Aug 14, 2026
Draft
@NathanColosimo NathanColosimo mentioned this pull request Aug 14, 2026
Open
@NathanColosimo NathanColosimo mentioned this pull request Aug 14, 2026
Merged
@NathanColosimo
NathanColosimo force-pushed the codex/replay-prune-runtime-graph branch from 7f68db4 to 4a2cc23 Compare August 14, 2026 08:08
@github-actions

github-actions Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

❌ Some tests failed

❌ Failed E2E Tests

▲ Vercel Production (8 failed)

python-node (8 failed):

  • promiseAllWorkflow | wrun_41M0VDN06C0GJF9R48YSG9WQ30 | 🔍 observability
  • sleepingWorkflow | wrun_41M0VDNNN70GXQ8MQGB8DFFBVP | 🔍 observability
  • parallelSleepWorkflow | wrun_41M0VDNQ4R0GTSQZ0NV78PS68F | 🔍 observability
  • nullByteWorkflow | wrun_41M0VDNXQD0GN74C8N2SH970J4 | 🔍 observability
  • cancelRun - cancelling a running workflow | wrun_41M0VDTG7Y0GKNVYEMFAZHAT66 | 🔍 observability
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0VDTH470GR3A7GYZW28QRRT | 🔍 observability
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0VDTWJP0GGDVNP9TXTDB25Y | 🔍 observability
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0VDVECE0GW8TM4DBWV2H88X | 🔍 observability

🌐 Cross-language Conformance (9 failed)

python (9 failed):

  • deploymentId: 'latest' is a no-op in non-Vercel worlds | wrun_01M0VDY8HAYKF5H611G7HRPJGV
  • promiseAllWorkflow | wrun_41M0VDN06C0GJF9R48YSG9WQ30
  • sleepingWorkflow | wrun_41M0VDNNN70GXQ8MQGB8DFFBVP
  • parallelSleepWorkflow | wrun_41M0VDNQ4R0GTSQZ0NV78PS68F
  • nullByteWorkflow | wrun_41M0VDNXQD0GN74C8N2SH970J4
  • cancelRun - cancelling a running workflow | wrun_41M0VDTG7Y0GKNVYEMFAZHAT66
  • cancelRun via CLI - cancelling a running workflow | wrun_41M0VDTH470GR3A7GYZW28QRRT
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_41M0VDTWJP0GGDVNP9TXTDB25Y
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_41M0VDVECE0GW8TM4DBWV2H88X

⚠️ Flaky E2E Tests (passed on retry)

These tests failed at least once and passed on a retry. A recurring entry here is a real race worth investigating.

  • cancelRun via CLI - cancelling a running workflow (nextjs-turbopack)
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered (express)

🛠 Infra Events (absorbed by the harness)

Platform anomalies the e2e harness detected and worked around (e.g. a run the queue never picked up, replaced by a fresh run). Clustered timestamps indicate a backend blip; a steady drip indicates a platform issue worth escalating.

34 infra events
  • cold-start-warmup · suite warmup (python) · at 02:59:19Z · abandoned wrun_41M0VDKMMW0GH51Y8JG5NNFA53 · (+7 more)
  • run-pickup-stall · sleepingWorkflow (python) · at 02:59:35Z · abandoned wrun_41M0VDQ9BQ0GQK2NWX2GHQEVNE
  • run-pickup-stall · parallelSleepWorkflow (python) · at 02:59:35Z · abandoned wrun_41M0VDQ9BR0GJ9VX6F75YE9E9Q
  • run-pickup-stall · nullByteWorkflow (python) · at 02:59:36Z · abandoned wrun_41M0VDQ9BV0GYFB5RFCSNMCKA8
  • run-pickup-stall · promiseAllWorkflow (python) · at 02:59:36Z · abandoned wrun_41M0VDQ9BH0GK9EQXW0CP2ZJCA
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 02:59:36Z · abandoned wrun_41M0VDQ9WK0GMVHTNXHTJD781E
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 03:00:09Z · abandoned wrun_41M0VDR9R80GNBR3X3RRSVHN07
  • run-pickup-stall · nullByteWorkflow (python) · at 03:00:36Z · abandoned wrun_41M0VDS4QY0GH9G06DD4CNPHN0
  • run-pickup-stall · promiseAllWorkflow (python) · at 03:00:36Z · abandoned wrun_41M0VDS50Q0GX4S6T0PQTXW632
  • run-pickup-stall · sleepingWorkflow (python) · at 03:00:36Z · abandoned wrun_41M0VDS50T0GSG4ZEE82KCJB3M
  • run-pickup-stall · parallelSleepWorkflow (python) · at 03:00:36Z · abandoned wrun_41M0VDS4WM0GM7PK0RSWPBVGA0
  • cold-start-warmup · suite warmup (tanstack-start) · at 03:00:41Z · abandoned wrun_01M0VDS3CBPAJR0ZH8W972WZZJ
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 03:01:27Z · abandoned wrun_41M0VDS9XE0GTMQHS9QWWG5A2J
  • cold-start-warmup · suite warmup (python) · at 03:01:53Z · abandoned wrun_01M0VDRA3CVH1MKDJEYP3V2BDW · (+7 more)
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 03:02:08Z · abandoned wrun_01M0VDVZ8CVJ5HP9G5FB1A1M3F
  • run-pickup-stall · sleepingWorkflow (python) · at 03:02:08Z · abandoned wrun_01M0VDVZ8H5BFT2BTQGSX5M1FZ
  • run-pickup-stall · parallelSleepWorkflow (python) · at 03:02:08Z · abandoned wrun_01M0VDVZ8M8XKZTWA23C6T2N7A
  • run-pickup-stall · nullByteWorkflow (python) · at 03:02:08Z · abandoned wrun_01M0VDVZ8NTRCFXWR32772DDWJ
  • run-pickup-stall · promiseAllWorkflow (python) · at 03:02:08Z · abandoned wrun_01M0VDVZ8DP1T84GQYSC3FST3V
  • run-pickup-stall · workflow throws after awaited setAttributes: attribute still persists on the failed run (nuxt) · at 03:03:00Z · abandoned wrun_01M0VDXJ26475QRTXK3E4NYG5E
  • run-pickup-stall · deploymentId: 'latest' is a no-op in non-Vercel worlds (python) · at 03:03:08Z · abandoned wrun_01M0VDXSW1STJBAFKY2ZTCJK9G
  • run-pickup-stall · sleepingWorkflow (python) · at 03:03:08Z · abandoned wrun_01M0VDXSW5NENDADCH34B3A56A
  • run-pickup-stall · promiseAllWorkflow (python) · at 03:03:08Z · abandoned wrun_01M0VDXSW3FS334GPAR9QK29DK
  • run-pickup-stall · nullByteWorkflow (python) · at 03:03:08Z · abandoned wrun_01M0VDXSWE2XMMPSNBC595YAX9
  • run-pickup-stall · parallelSleepWorkflow (python) · at 03:03:08Z · abandoned wrun_01M0VDXSW8ATGQPPHSTAPYGT45
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 03:03:08Z · abandoned wrun_41M0VDW1SZ0GY2PX4NR4R4Y8X5
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 03:03:08Z · abandoned wrun_41M0VDVWMG0GJ7M15MXRX9SJEM
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 03:04:08Z · abandoned wrun_01M0VDZMFMRD0Y1PD7NBT2M72C
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 03:04:08Z · abandoned wrun_01M0VDZMFVFN8Q4FA149CMQ9EQ
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 03:04:08Z · abandoned wrun_01M0VDZMFZ1SM0TW62J5SAFBK3
  • run-pickup-stall · cancelRun via CLI - cancelling a running workflow (python) · at 03:04:38Z · abandoned wrun_01M0VE0HVJRVGZC5NAGY22430N
  • run-pickup-stall · cancelRun - cancelling a running workflow (python) · at 03:04:38Z · abandoned wrun_01M0VE0HVDHGJS57WFKR4Y7XDS
  • run-pickup-stall · sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration (python) · at 03:05:08Z · abandoned wrun_01M0VE1F35J38MPSCFZMN6YX2P
  • run-pickup-stall · WorkflowNotRegisteredError fails the run when workflow does not exist (nextjs-webpack) · at 03:05:16Z · abandoned wrun_01M0VE1QKFX8AWRVN1GKHDE0K9

E2E Test Summary

Summary
Passed Failed Skipped Total
❌ ▲ Vercel Production 3570 8 742 4320
✅ 💻 Local Development 3922 0 558 4480
✅ 📦 Local Production 3922 0 558 4480
✅ 🐘 Local Postgres 3922 0 558 4480
✅ 🪟 Windows 320 0 0 320
❌ 🌐 Cross-language Conformance 0 9 132 141
✅ vercel-http-transport 817 0 143 960
✅ vercel-multi-region 27 0 0 27
✅ vercel-ws-transport 553 0 87 640
Total 17053 17 2778 19848
Details by Category

❌ ▲ Vercel Production

App Passed Failed Skipped
✅ astro-node 132 0 28
✅ astro-quickjs 132 0 28
✅ example-node 132 0 28
✅ example-quickjs 132 0 28
✅ express-node 132 0 28
✅ express-quickjs 132 0 28
✅ fastify-node 132 0 28
✅ fastify-quickjs 132 0 28
✅ hono-node 132 0 28
✅ hono-quickjs 132 0 28
✅ nest-node 132 0 28
✅ nest-quickjs 132 0 28
✅ nextjs-turbopack-node 157 0 3
✅ nextjs-turbopack-quickjs 157 0 3
✅ nextjs-webpack-node 157 0 3
✅ nextjs-webpack-quickjs 157 0 3
✅ nitro-node 132 0 28
✅ nitro-quickjs 132 0 28
✅ nuxt-node 132 0 28
✅ nuxt-quickjs 132 0 28
❌ python-node 0 8 152
✅ sveltekit-node 151 0 9
✅ sveltekit-quickjs 151 0 9
✅ tanstack-start-node 132 0 28
✅ tanstack-start-quickjs 132 0 28
✅ vite-node 132 0 28
✅ vite-quickjs 132 0 28

✅ 💻 Local Development

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 📦 Local Production

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 🐘 Local Postgres

App Passed Failed Skipped
✅ astro-stable-node 134 0 26
✅ astro-stable-quickjs 134 0 26
✅ express-stable-node 134 0 26
✅ express-stable-quickjs 134 0 26
✅ fastify-stable-node 134 0 26
✅ fastify-stable-quickjs 134 0 26
✅ hono-stable-node 134 0 26
✅ hono-stable-quickjs 134 0 26
✅ nest-stable-node 134 0 26
✅ nest-stable-quickjs 134 0 26
✅ nextjs-turbopack-canary-node 141 0 19
✅ nextjs-turbopack-canary-quickjs 141 0 19
✅ nextjs-turbopack-stable-node 160 0 0
✅ nextjs-turbopack-stable-quickjs 160 0 0
✅ nextjs-webpack-canary-node 141 0 19
✅ nextjs-webpack-canary-quickjs 141 0 19
✅ nextjs-webpack-stable-node 160 0 0
✅ nextjs-webpack-stable-quickjs 160 0 0
✅ nitro-stable-node 134 0 26
✅ nitro-stable-quickjs 134 0 26
✅ nuxt-stable-node 134 0 26
✅ nuxt-stable-quickjs 134 0 26
✅ sveltekit-stable-node 153 0 7
✅ sveltekit-stable-quickjs 153 0 7
✅ tanstack-start-node 134 0 26
✅ tanstack-start-quickjs 134 0 26
✅ vite-stable-node 134 0 26
✅ vite-stable-quickjs 134 0 26

✅ 🪟 Windows

App Passed Failed Skipped
✅ nextjs-turbopack-node 160 0 0
✅ nextjs-turbopack-quickjs 160 0 0

❌ 🌐 Cross-language Conformance

App Passed Failed Skipped
❌ python 0 9 132

✅ vercel-http-transport

App Passed Failed Skipped
✅ example 132 0 28
✅ express 132 0 28
✅ hono 132 0 28
✅ nextjs-turbopack 157 0 3
✅ nitro 132 0 28
✅ vite 132 0 28

✅ vercel-multi-region

App Passed Failed Skipped
✅ nextjs-turbopack 27 0 0

✅ vercel-ws-transport

App Passed Failed Skipped
✅ example 132 0 28
✅ express 132 0 28
✅ nextjs-turbopack 157 0 3
✅ vite 132 0 28

📋 View full workflow run

@github-actions

github-actions Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 4834383 · Tue, 25 Aug 2026 03:22:33 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 240 (-38%) 💚 1525 🔴 (+39%) 🔻 1574 🔴 (+39%) 🔻 1838 🔴 (+32%) 🔻 30
TTFS stream 485 (+177%) 🔻 1484 🔴 (+34%) 🔻 1516 🔴 (+34%) 🔻 1567 🔴 (+36%) 🔻 30
TTFS hook + stream 1712 (+199%) 🔻 1839 🔴 (+35%) 🔻 1897 🔴 (+20%) 🔻 1990 🔴 (-57%) 💚 30
Fan-out TTFS Promise.all(100 steps) 640 (+2.2%) 2135 (+95%) 🔻 2156 (+39%) 🔻 2292 (-54%) 💚 10
Fan-out TTLS Promise.all(100 steps) 2612 (+44%) 🔻 4012 (±0%) 4044 (-0.7%) 10348 (+3.9%) 10
STSO 1020 steps (inline) 100 (-11%) 156 (+9.1%) 175 (+5.4%) 281 (+17%) 🔻 1019
WO 1020 steps 157552 (+8.3%) 157552 (+8.3%) 157552 (+8.3%) 157552 (+8.3%) 1
CRTT first chunk (pooled) 125 (+51%) 🔻 198 (+41%) 🔻 246 (-5.4%) 458 (-35%) 💚 28

Streams

Scenario CRTT 1st p75 p90 p99 CDV max iters
paced control (100/s, 60B) 154 (+38%) 192 (+27%) 257 (-30%) 390 (-42%) 162 (+33%) 10
size sweep (100/s, 160B-12KB) 147 (+31%) 197 (+55%) 323 (+99%) 1023 (+56%) 228 (+111%) 10
replay gateway-gpt-5.4-nano-2000t (1x) 182 (+2%) 146 (-77%) 176 (-92%) 303 (-92%) 211 (-46%) 3
replay eve-gpt-5.6-sol-2000t (1x) 206 (+49%) 163 (+16%) 216 (+14%) 379 (-26%) 221 (-56%) 2
replay eve-gpt-5.6-sol-2000t (2x) 167 (+18%) 208 (+11%) 286 (+6%) 673 (-19%) 290 (+60%) 3
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 144386ms → this run 156134ms (Δ +11748ms, +8%)

100-150 ms  █████████████████┃██████  main 853  this 646  -207
150-200 ms  ████░░░░┃                 main 138  this 336  +198
200-250 ms  ┃                         main  19  this  22    +3
250-300 ms  ┃                         main   4  this   7    +3
300-350 ms  ┃                         main   5  this   3    -2
350-400 ms  ┃                         main   0  this   2    +2
400-450 ms  ┃                         main   0  this   2    +2
600-650 ms  ┃                         main   0  this   1    +1
📈 CRTT drill-down vs main (RTT distributions & profiles) 
variant  RTT 1ms→5s+             avg         p50         p90          p99     n
control  ······▂█▂····  150.5 (+19%)  135 (+27%)  257 (-30%)   390 (-42%)  3000
sweep    ······▂█▂▁▁··  172.3 (+46%)  145 (+46%)  323 (+99%)  1023 (+56%)  3000
gw 1x    ······▃█▁····    129 (-56%)   125 (+2%)  176 (-92%)   303 (-92%)  5295
eve 1x   ······▃█▂····  138.3 (+14%)  125 (+23%)  216 (+14%)   379 (-26%)  5186
eve 2x   ······▁█▃▁···  171.6 (+16%)  156 (+16%)   286 (+6%)   673 (-19%)  7779

RTT over stream progress (avg per tenth of stream, bars scaled min→max):

control  █▇▄▃▂▃▃▁▁▁  130–189ms
sweep    ▃██▅▃▂▁▂▁▂  133–242ms
gw 1x    ▅▄▄█▄▃▂▂▃▁  120–144ms
eve 1x   ▆▂▁▃▁▁██▂▃  124–166ms
eve 2x   ▄▁▃▂▂▆▃▅▆█  145–207ms

RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max):

sweep  █▄▄▄▃▁▄  170–175ms

Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max):

control  ▁▅▃█▄█▄▂▂▁  40–50ms
sweep    ▁▆▅▄▃▆▄▆▃█  48–65ms
gw 1x    ▄▆▃▁▅▂▅█▁▃  38–43ms
eve 1x   ▇▄▁▄▃▃▆▃▂█  23–32ms
eve 2x   █▇▁▂▆▅▄▁▃▆  24–33ms
ℹ️ Metric definitions & methodology

Streams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach.

The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). █ = main, ┃ = this run, ░ = fill.

The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, · = empty) and mean RTT/positive-CDV profile lines over stream progress and chunk size. Histograms, avgs, and profiles merge exactly across runs; p50–p99 are percentile-of-percentiles. Per-index rows live in the artifacts.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles

Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t eaf22f5946e7c61f3c65c7006d550df180cfabd4e706254a09f22aec0cfb420d · gateway-gpt-5.4-nano-2000t 6f24ac518b6b83ff1d0e85a5fe78230db192716d66a7fc6b2fe022752001d041

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600

All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = start() → first step body (includes dispatch + any cold start); Fan-out TTFS/TTLS = first/last step completion of one Promise.all from the same anchor (the gap is the runtime’s fan-out spread); STSO/WO between step bodies; CRTT inside the workflow (excludes the api.vercel.com read path).

Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor.

@NathanColosimo
NathanColosimo force-pushed the codex/replay-prune-runtime-graph branch from 8edf496 to 35bea10 Compare August 14, 2026 21:06
VaguelySerious
VaguelySerious approved these changes Aug 21, 2026
View reviewed changes

@VaguelySerious VaguelySerious left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: no blocking issues

Comment thread packages/builders/src/workflow-bundle-boundary.test.ts Outdated
Comment thread packages/world/src/index.ts
Comment thread packages/world/src/attributes-validation.ts
Comment thread packages/world/src/event-metadata.ts Outdated
TooTallNate
TooTallNate approved these changes Aug 24, 2026
View reviewed changes

@TooTallNate TooTallNate left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI: Approved. I found no blocking issues in the schema-free module split, validation changes, package exports, or bundle-boundary coverage.

@github-actions

github-actions Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

No backport to stable for d62b444 (AI decision).

This is primarily a bundle-size/startup-performance refactor that moves World validation and event-metadata helpers onto new @workflow/world subpath exports and intentionally removes public root-level exports (validateAttributeKey, validateAttributeValue, EVENT_DATA_PAYLOAD_FIELD_BY_EVENT_TYPE, EVENT_DATA_REF_FIELDS) as a deliberate v5 breaking change, carrying a major changeset — exactly the kind of API-surface churn a maintenance line should not take. The two genuine hardening bits (an unknown attribute deletion no longer offsetting a newly added key in the per-run cap accounting in validateAttributeChanges, and prototype-like event type strings such as constructor returning no metadata) are small and could be split out and force-backported on their own if desired. Per the mixed-commit rule, the presence of a breaking refactor makes the whole commit ineligible.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

d62b44473b43e183e71386fe84b33f5e7bb5445c

This was referenced Aug 25, 2026
Merged
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@TooTallNate TooTallNate TooTallNate approved these changes
@VaguelySerious VaguelySerious VaguelySerious approved these changes
@msullivan msullivan Awaiting requested review from msullivan
@fantix fantix Awaiting requested review from fantix

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@NathanColosimo @TooTallNate @VaguelySerious