Default the events transport to WebSockets - #3702
Conversation
🦋 Changeset detectedLatest commit: 5c3f965 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (1 failed)nuxt-node (1 failed):
💻 Local Development (1 failed)nextjs-webpack-stable-node (1 failed):
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ❌ ▲ Vercel Production | 3577 | 1 | 742 | 4320 |
| ❌ 💻 Local Development | 3921 | 1 | 558 | 4480 |
| ✅ 📦 Local Production | 3922 | 0 | 558 | 4480 |
| ✅ 🐘 Local Postgres | 3922 | 0 | 558 | 4480 |
| ✅ 🪟 Windows | 320 | 0 | 0 | 320 |
| ✅ 🌐 Cross-language Conformance | 9 | 0 | 132 | 141 |
| ✅ vercel-http-transport | 817 | 0 | 143 | 960 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| ✅ vercel-ws-transport | 553 | 0 | 87 | 640 |
| Total | 17068 | 2 | 2778 | 19848 |
Details by Category
❌ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 132 | 0 | 28 |
| ✅ astro-quickjs | 132 | 0 | 28 |
| ✅ example-node | 132 | 0 | 28 |
| ✅ example-quickjs | 132 | 0 | 28 |
| ✅ express-node | 132 | 0 | 28 |
| ✅ express-quickjs | 132 | 0 | 28 |
| ✅ fastify-node | 132 | 0 | 28 |
| ✅ fastify-quickjs | 132 | 0 | 28 |
| ✅ hono-node | 132 | 0 | 28 |
| ✅ hono-quickjs | 132 | 0 | 28 |
| ✅ nest-node | 132 | 0 | 28 |
| ✅ nest-quickjs | 132 | 0 | 28 |
| ✅ nextjs-turbopack-node | 157 | 0 | 3 |
| ✅ nextjs-turbopack-quickjs | 157 | 0 | 3 |
| ✅ nextjs-webpack-node | 157 | 0 | 3 |
| ✅ nextjs-webpack-quickjs | 157 | 0 | 3 |
| ✅ nitro-node | 132 | 0 | 28 |
| ✅ nitro-quickjs | 132 | 0 | 28 |
| ❌ nuxt-node | 131 | 1 | 28 |
| ✅ nuxt-quickjs | 132 | 0 | 28 |
| ✅ python-node | 8 | 0 | 152 |
| ✅ sveltekit-node | 151 | 0 | 9 |
| ✅ sveltekit-quickjs | 151 | 0 | 9 |
| ✅ tanstack-start-node | 132 | 0 | 28 |
| ✅ tanstack-start-quickjs | 132 | 0 | 28 |
| ✅ vite-node | 132 | 0 | 28 |
| ✅ vite-quickjs | 132 | 0 | 28 |
❌ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ❌ nextjs-webpack-stable-node | 159 | 1 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-quickjs | 160 | 0 | 0 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 9 | 0 | 132 |
✅ vercel-http-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 132 | 0 | 28 |
| ✅ express | 132 | 0 | 28 |
| ✅ hono | 132 | 0 | 28 |
| ✅ nextjs-turbopack | 157 | 0 | 3 |
| ✅ nitro | 132 | 0 | 28 |
| ✅ vite | 132 | 0 | 28 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
✅ vercel-ws-transport
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ example | 132 | 0 | 28 |
| ✅ express | 132 | 0 | 28 |
| ✅ nextjs-turbopack | 157 | 0 | 3 |
| ✅ vite | 132 | 0 | 28 |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 250068ms → this run 123169ms (Δ -126899ms, -51%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): 📜 Previous results (2)9a9fbabFri, 21 Aug 2026 23:35:04 GMT · run logs
Streams
6d0d5afFri, 21 Aug 2026 20:24:38 GMT · run logs
Streams
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
There was a problem hiding this comment.
Additional Suggestions:
- Docs state
WORKFLOW_EVENTS_TRANSPORTdefault ishttp, but the code now defaults to WebSocket unless the value is exactlyhttp.
isWsEventsTransportEnabledunit tests still assert the old opt-in defaults, so they fail after the transport default flipped to WS (!== 'http').
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
|
Both bot suggestions are fixed. e2e failures look preexisting on main. |
a8726db to
6d0d5af
Compare
WORKFLOW_EVENTS_TRANSPORT=http is the opt-out. Only that exact value disables it, so a typo'd or empty value fails toward the default rather than quietly pinning a deployment to HTTP. The prerequisite the gate named for defaulting on is met: postEventFrameOverWs opens a client span per frame. What is still missing is Vercel's outgoing-requests view, which reads instrumented fetch calls rather than spans and so cannot show a transport that issues no request. Co-Authored-By: opencode <opencode@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
Three places still documented http as the default. Each now states the opt-out is the exact value http, rather than leaving 'default: ws' to imply that anything non-ws disables it — the asymmetry is deliberate in the code and is the part a reader would otherwise get wrong. Also drops 'Experimental' from the Vercel World page: a setting that is on for everyone by default is not opt-in experimental, whatever else it is. Co-Authored-By: opencode <opencode@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
Five tests in ws-transport.test.ts still encoded the opt-in semantics. Three were the isWsEventsTransportEnabled table itself; the other two (openWsChannel 'does nothing when the gate is off', and the channel release equivalent) relied on the suite's ambient unset environment meaning 'off', which it no longer does. Both now set http explicitly. Two tests in ws-transport-spans.test.ts asserted HTTP-side span behaviour the same way. The write one would have kept passing by falling through resolveWsTransport's null rather than because the gate was off - passing for the wrong reason, which is what this file exists to catch. Also makes the opt-out case-insensitive and trimmed. The gate is deliberately asymmetric - unrecognized values take the default - but that asymmetry should not extend to swallowing HTTP or ' http '. Whoever reaches for the escape hatch is plausibly mid-incident, and silently ignoring their opt-out over a capital letter is the same class of silent-wrong-transport bug this flip is meant to stop shipping. 554 tests pass in packages/world-vercel. Co-Authored-By: opencode <opencode@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
Flipping the default makes e2e-vercel-prod a WebSocket lane: it sets no WORKFLOW_EVENTS_TRANSPORT, and unset now means ws. Nothing in the file would exercise the HTTP events transport against a real deployment any more, so this is not additive coverage — it replaces coverage the flip silently removed. Unconditional and required rather than label-gated like the WS lane. HTTP is now the fallback, and the fallback is silent: resolveWsTransport returning null costs a write nothing and logs nothing, which is the shape of the durabench bug this stack came out of. Two apps rather than the WS lane's four, since every row is a real vercel deploy charged to every PR. nextjs-turbopack is the only fixture emitting OTEL spans, so it is the one that can show which transport actually ran; express covers the non-Next server path. Also corrects the WS lane's docblock, which claimed every other job exercises HTTP only. That stopped being true one commit ago. Co-authored-by: vercel[bot] <35613825+vercel[bot]@users.noreply.github.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
The WS e2e lane asserts that the transport is harmless, not that it is used: an event written over HTTP produces the same run outcome as one written over the socket, so the lane stayed green through the entire period the transport was silently demoted. WORKFLOW_INTERNAL_EVENTS_TRANSPORT_STRICT turns that one case into a failed run, and the WS lane now sets it. Scoped to step_completed alone, because most fallback is legitimate: run_created is written outside any invocation that opens a channel; run_started routinely lands before the channel is registered (34% HTTP on a healthy deployment); step_created and wait_created mostly fold into events.createBatch, which is not wired to the socket; and a write after the invocation released its claim falls back by design. step_completed is issued after a step body has run, and was 100% ws across every WS-enabled deployment measured on two SDK versions. The flag reads as off unless the value is exactly 1 or true - the opposite asymmetry from the transport gate, which treats an unrecognized value as on. That gate risks a deployment sitting quietly on the wrong transport; this one fails runs, and should not be acquired by a typo. Co-Authored-By: opencode <opencode@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
It was opt-in behind ws-transport-test because four real vercel deploys were too much to charge an unrelated PR for a transport that was off by default. Flipping the default expires that reasoning from both ends: the cost is no longer for someone else's feature, and this is now the only lane that asserts the socket carried the events. e2e-vercel-prod inherits the new default but checks nothing, so behind a label the average PR would move every deployment onto WebSockets with nothing verifying they were used. Drops WS_REQUIRED from the gate along with it. That existed only to let the lane be legitimately skipped on an unlabelled PR; with no label the lane is required unconditionally, like e2e-vercel-prod and the HTTP lane, and the skipped case is now a failure rather than a warning. Gate script extracted and run against the cases that matter: ws skipped fails on a standard PR, ws skipped fails under workflow-server-test, and all-green passes. Co-Authored-By: opencode <opencode@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
Before the flip, HTTP was the default and all 28 e2e-vercel-prod lane-runs covered it. After the flip they cover WebSockets instead, and this lane is the entirety of the HTTP coverage - two apps was too thin for a transport that is still supported. Six, not the full 14, because every row is a real vercel deploy charged to every PR. Chosen by server shape rather than count: example (baseline), nextjs-turbopack (Next, and the only fixture emitting OTEL spans), vite (Vite SSR), express (Node req/res), nitro (h3, also covers nuxt) and hono (fetch-API Request/Response, a different mount shape from express). The rest duplicate a shape already covered; python is left out because it has no conformance gate and needs routes this suite does not serve. The first four match the WS lane's matrix on purpose, so the same fixture runs on both transports and a failure on one can be read against the other. Project ids and slugs are copied from e2e-vercel-prod and verified equal to it; both lanes already use the same team and token. Co-Authored-By: opencode <opencode@vercel.com> Co-Authored-By: shalabhc <shalabh.chaturvedi@vercel.com>
|
No backport to This flips the default of To override, re-run the Backport to stable workflow manually via |
2 participants
We have decided to make websockets opt-out in v5.
Flip websockets to default
on. It can be disabled by settingWORKFLOW_EVENTS_TRANSPORT=httpAlso:
Known issues
Tests
All existing e2e tests will now test websockets on by default.
This leaves a large gap for http coverage while we we want to keep a safe fallback. I'm adding a separate set of parallel tests to tests all e2e for http - separate PR.