Repository navigation
fix(fastify): guard request pump against re-entry (#1824) - #1932
Merged
Merged
Conversation
js_fastify_process_pending dispatches each request handler inside a setjmp try-frame (call_closure2_catching -> js_try_push). A handler that awaits pumps the event loop (js_async_first_call -> js_promise_run_microtasks -> js_stdlib_process_pending), which re-enters js_fastify_process_pending while the outer handler's try-frame is still open on the native stack. The re-entrant call started another request handler, whose try-frame nested inside the suspended outer one. Under concurrent load the nesting grew one level per in-flight request and blew MAX_TRY_DEPTH (128), aborting with 'Try block nesting too deep' — observed as a silent SIGSEGV (exit 139) under PM2 on Linux x86_64. Add a thread-local re-entrancy guard: a nested pump call processes no new requests and returns 0. Pending requests stay queued in each server's request_rx and are picked up on the next top-level pump tick once the current handler unwinds its try-frame. The in-flight handler's own await still makes progress — its resolution drains through PENDING_RESOLUTIONS in js_stdlib_process_pending, not through this request loop. Repro (pre-fix): a Fastify + @perryts/mysql server crashes within the first few requests under concurrent load (e.g. 500 reqs @ concurrency 30); sequential load never crashes because requests never overlap. Post-fix: 5000 reqs @ concurrency 100 complete cleanly, server stays up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1824 — a compiled Fastify +
@perryts/mysqlservice SIGSEGVs (exit 139, silent stderr, PM2 restart) under sustained load. The reporter saw it after ~260 sequentialawait fetch()+ mysqlexecute()calls; the same fault reproduces locally in chirp (Fastify +@perryts/mysql, no fetch at all), which pinned the real cause to the Fastify request pump rather than to fetch or GC.Root cause
js_fastify_process_pendingdispatches each request handler inside a setjmp try-frame (call_closure2_catching→js_try_push). A handler thatawaits pumps the event loop:The re-entrant pump call starts another request handler, whose try-frame nests inside the suspended outer one (still open on the native stack mid-
await). Under concurrent load the nesting grows one level per in-flight request and blowsMAX_TRY_DEPTH(128) incrates/perry-runtime/src/exception.rs, aborting withTry block nesting too deep. On Linux x86_64 under PM2 this surfaces as a silent SIGSEGV (exit 139) — which is why it looked like #1597 / a GC bug, and whyPERRY_GEN_GC=0,PERRY_GEN_GC_EVACUATE=0, #1715 and #1825 all left it in place.This is why sequential load never crashed (requests never overlap, so try-frames never stack) but concurrent load does.
Fix
A thread-local re-entrancy guard on
js_fastify_process_pending: a nested pump call processes no new requests and returns 0. Pending requests stay queued in each server'srequest_rxand are picked up on the next top-level pump tick, once the current handler unwinds its try-frame. The in-flight handler's ownawaitstill makes progress — its resolution drains throughPENDING_RESOLUTIONSinjs_stdlib_process_pending, not through this request loop.The sibling pump
js_http_process_pending(node:http) dispatches handlers viajs_closure_call0/1without a try-frame wrapper, so it does not accumulate try depth and is unaffected.Validation
Minimal Fastify +
@perryts/mysqlserver (two awaited queries per request), local MySQL:Try block nesting too deep) within the first few requestscargo fmt --all -- --check: cleancargo test --release -p perry-stdlib: 79 passed, 0 failed