[e2e] Decay the repro's poke pump instead of stopping at the budget - #3561
Conversation
A hard ceiling bought the lanes their green at the cost of coverage: every step-storm run on both local lanes spent the full 64-poke budget, so the back half of a ~160s run applied no out-of-band writes at all. Lowering attempt concurrency does not help — at c3 runs finish in 87-96s and still spend the whole budget. The pump now slows to `pokeIntervalMs * POKE_DECAY_FACTOR` (8) once the budget is spent rather than stopping, so loop gain stays below 1 while a slow run's later rounds keep receiving pressure. Verified locally at POKE_MAX=4 / 500ms / decay 10: a 26.1s run sent 8 pokes, 4 at full cadence and 4 across the remaining 24s, where unbounded sends ~52 and a hard stop sends 4. The lanes were never comparable on this axis to begin with: a Vercel resume pays a network round trip, so that lane's pump achieves an effective ~2.3s interval (35-44 pokes per run) while localhost runs the full 750ms. The decayed rate is what puts the local lanes near the lane that found the production bug, rather than at 3x its write rate. Also records what the saturated lanes are NOT: world-local logged zero failed deliveries, zero handler errors and zero exhausted messages across three 14-run passes, because its semaphore parks a message before the delivery fetch and queue waiting never consumes the transport timeout. No retry amplification to find. The two default-concurrency smells that did surface (world-local's 1000 in-flight deliveries, world-postgres' 50 workers per process) are noted for their own change, not fixed here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Peter Wielander <peter.wielander@vercel.com>
With the pump's pressure restored, world-local `step-storm` lands at 194-203s and world-postgres at 168-175s against the harness' 240000 default — 83% of their own timeout — while the Vercel lane finishes the same storm in 85-100s. One slow runner at that margin turns a lane that reproduces into a lane full of `stuck`, which is the failure this branch exists to stop. The local script raises it to 480000. It is the one scale knob the script sets, because it is the one whose meaning depends on every replay of every run sharing a Node process rather than getting its own Fluid invocation; an exported value still wins, and a workflow_dispatch input still wins. Evidence the restored pressure is worth the timeout: on the decay config world-local produced a CORRUPTED_EVENT_LOG (wrun_01M00SW88YV76AXKXQFX4Y3V1X, 87 pokes over 200s), which three passes of the hard-cap config never did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Peter Wielander <peter.wielander@vercel.com>
Signed-off-by: Peter Wielander <peter.wielander@vercel.com>
🦋 Changeset detectedLatest commit: 236253d The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (251 failed)astro-node (8 failed):
astro-quickjs (12 failed):
example-node (10 failed):
example-quickjs (12 failed):
express-node (7 failed):
express-quickjs (11 failed):
fastify-node (7 failed):
fastify-quickjs (12 failed):
hono-node (9 failed):
hono-quickjs (10 failed):
nest-node (12 failed):
nest-quickjs (14 failed):
nextjs-turbopack-node (5 failed):
nextjs-turbopack-quickjs (10 failed):
nextjs-webpack-node (10 failed):
nextjs-webpack-quickjs (8 failed):
nitro-node (8 failed):
nitro-quickjs (9 failed):
nuxt-node (10 failed):
nuxt-quickjs (10 failed):
python-node (8 failed):
sveltekit-node (8 failed):
sveltekit-quickjs (12 failed):
tanstack-start-node (6 failed):
tanstack-start-quickjs (5 failed):
vite-node (7 failed):
vite-quickjs (11 failed):
🪟 Windows (1 failed)nextjs-turbopack-quickjs (1 failed):
🌐 Cross-language Conformance (9 failed)python (9 failed):
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ❌ ▲ Vercel Production | 3327 | 251 | 742 | 4320 |
| ✅ 💻 Local Development | 3922 | 0 | 558 | 4480 |
| ✅ 📦 Local Production | 3922 | 0 | 558 | 4480 |
| ✅ 🐘 Local Postgres | 3922 | 0 | 558 | 4480 |
| ❌ 🪟 Windows | 319 | 1 | 0 | 320 |
| ❌ 🌐 Cross-language Conformance | 0 | 9 | 132 | 141 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| Total | 15439 | 261 | 2548 | 18248 |
Details by Category
❌ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ astro-node | 124 | 8 | 28 |
| ❌ astro-quickjs | 120 | 12 | 28 |
| ❌ example-node | 122 | 10 | 28 |
| ❌ example-quickjs | 120 | 12 | 28 |
| ❌ express-node | 125 | 7 | 28 |
| ❌ express-quickjs | 121 | 11 | 28 |
| ❌ fastify-node | 125 | 7 | 28 |
| ❌ fastify-quickjs | 120 | 12 | 28 |
| ❌ hono-node | 123 | 9 | 28 |
| ❌ hono-quickjs | 122 | 10 | 28 |
| ❌ nest-node | 120 | 12 | 28 |
| ❌ nest-quickjs | 118 | 14 | 28 |
| ❌ nextjs-turbopack-node | 152 | 5 | 3 |
| ❌ nextjs-turbopack-quickjs | 147 | 10 | 3 |
| ❌ nextjs-webpack-node | 147 | 10 | 3 |
| ❌ nextjs-webpack-quickjs | 149 | 8 | 3 |
| ❌ nitro-node | 124 | 8 | 28 |
| ❌ nitro-quickjs | 123 | 9 | 28 |
| ❌ nuxt-node | 122 | 10 | 28 |
| ❌ nuxt-quickjs | 122 | 10 | 28 |
| ❌ python-node | 0 | 8 | 152 |
| ❌ sveltekit-node | 143 | 8 | 9 |
| ❌ sveltekit-quickjs | 139 | 12 | 9 |
| ❌ tanstack-start-node | 126 | 6 | 28 |
| ❌ tanstack-start-quickjs | 127 | 5 | 28 |
| ❌ vite-node | 125 | 7 | 28 |
| ❌ vite-quickjs | 121 | 11 | 28 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 134 | 0 | 26 |
| ✅ astro-stable-quickjs | 134 | 0 | 26 |
| ✅ express-stable-node | 134 | 0 | 26 |
| ✅ express-stable-quickjs | 134 | 0 | 26 |
| ✅ fastify-stable-node | 134 | 0 | 26 |
| ✅ fastify-stable-quickjs | 134 | 0 | 26 |
| ✅ hono-stable-node | 134 | 0 | 26 |
| ✅ hono-stable-quickjs | 134 | 0 | 26 |
| ✅ nest-stable-node | 134 | 0 | 26 |
| ✅ nest-stable-quickjs | 134 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 160 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 141 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 141 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 160 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 160 | 0 | 0 |
| ✅ nitro-stable-node | 134 | 0 | 26 |
| ✅ nitro-stable-quickjs | 134 | 0 | 26 |
| ✅ nuxt-stable-node | 134 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 134 | 0 | 26 |
| ✅ sveltekit-stable-node | 153 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 153 | 0 | 7 |
| ✅ tanstack-start-node | 134 | 0 | 26 |
| ✅ tanstack-start-quickjs | 134 | 0 | 26 |
| ✅ vite-stable-node | 134 | 0 | 26 |
| ✅ vite-stable-quickjs | 134 | 0 | 26 |
❌ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 160 | 0 | 0 |
| ❌ nextjs-turbopack-quickjs | 159 | 1 | 0 |
❌ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ❌ python | 0 | 9 | 132 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 182882ms → this run 199778ms (Δ +16896ms, +9%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): 📜 Previous results (1)5426321Fri, 14 Aug 2026 19:45:01 GMT · run logs
ℹ️ Metric definitions & methodologyStreams: first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 world-sim scenario book — 1 fail of 41 total
Full trace: |
Event Log Race Repro
Run History
Latest Non-Completed Runs
Config
|
#3696 added five blocked-branch dispatch inputs and wired them into the Vercel job only, so a dispatch tuning that scenario reached one lane and left the other two on harness defaults — the silent per-lane divergence the inputs' own comment warns about. Every EVENT_LOG_RACE_REPRO_* knob now reaches both jobs. Also names the fourth scenario in the AGENTS.md intro, which still said three. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Peter Wielander <peter.wielander@vercel.com>
# Conflicts: # AGENTS.md
|
No backport to This is tuning of the main-only event-log-race-repro investigation harness: it adds a new To override, re-run the Backport to stable workflow manually via |
Follow-up to #3558, which landed the poke budget as a hard stop.
step-stormrun spent all 64 pokes, so the back half of a ~160s run got no out-of-band writes. Dropping attempt concurrency to 3 doesn't help (runs finish in 87-96s and still spend it).pokeIntervalMs * POKE_DECAY_FACTOR(8) after the budget instead of stopping. Loop gain stays under 1, later rounds still get pressure.CORRUPTED_EVENT_LOGonce in 18step-stormruns (wrun_01M00SW88YV76AXKXQFX4Y3V1X, 87 pokes, 200s). Three passes of the merged config: zero.RUN_TIMEOUT_MSto 480000. Exported values and dispatch inputs still win.step-storm,hook-stormandhook-sleepclean on all three lanes, no stuck runs,step-stormat 148-172s locally against the raised timeout. Every corruption in the run isblocked-branch.poke budget 64 then /8).Merged
main, which added theblocked-branchscenario (#3696). That scenario corrupts by design and stays red until a fix lands, so the lanes on this PR are red for its reasons, not this one's. Check the per-scenario breakdown:step-stormis the one this PR moves.Two fixes the merge turned up:
blocked-branchinputs into the Vercel job only, so a dispatch tuning that scenario left the two local lanes on harness defaults. EveryEVENT_LOG_RACE_REPRO_*knob now reaches both jobs.Known gap, not fixed here: local
hook-stormstraggles ~1 branch per round by arithmetic (index * 400msstagger vs a 2500ms watchdog), where Vercel gets 18-24 because each resume also pays ~1.5s of network latency. Retuning the local stagger or watchdog needs its own straggler numbers.PR Checklist - Required to merge
pnpm changesetwas run to create a changelog for this PRgit commit --signoff)@vercel/workflowin a comment once the PR is ready, and the above checklist is complete🤖 Generated with Claude Code