Skip to content

fix(fastify): guard request pump against re-entry (#1824) - #1932

Merged
proggeramlug merged 1 commit into
mainfrom
worktree-fix-1824-fetch-mysql-segv
May 27, 2026
Merged

proggeramlug merged 1 commit into
mainfrom
worktree-fix-1824-fetch-mysql-segv

Conversation

@proggeramlug

Copy link
Copy Markdown
Contributor

Summary

Fixes #1824 — a compiled Fastify + @perryts/mysql service SIGSEGVs (exit 139, silent stderr, PM2 restart) under sustained load. The reporter saw it after ~260 sequential await fetch() + mysql execute() calls; the same fault reproduces locally in chirp (Fastify + @perryts/mysql, no fetch at all), which pinned the real cause to the Fastify request pump rather than to fetch or GC.

Root cause

js_fastify_process_pending dispatches each request handler inside a setjmp try-frame (call_closure2_catching → js_try_push). A handler that awaits pumps the event loop:

js_fastify_process_pending
  → call_closure2_catching (js_try_push, try_depth++)
    → js_async_first_call
      → js_promise_run_microtasks
        → js_stdlib_process_pending
          → js_fastify_process_pending   ← RE-ENTRANT
            → call_closure2_catching (js_try_push, try_depth++)
              → ... (repeats)

The re-entrant pump call starts another request handler, whose try-frame nests inside the suspended outer one (still open on the native stack mid-await). Under concurrent load the nesting grows one level per in-flight request and blows MAX_TRY_DEPTH (128) in crates/perry-runtime/src/exception.rs, aborting with Try block nesting too deep. On Linux x86_64 under PM2 this surfaces as a silent SIGSEGV (exit 139) — which is why it looked like #1597 / a GC bug, and why PERRY_GEN_GC=0, PERRY_GEN_GC_EVACUATE=0, #1715 and #1825 all left it in place.

This is why sequential load never crashed (requests never overlap, so try-frames never stack) but concurrent load does.

Fix

A thread-local re-entrancy guard on js_fastify_process_pending: a nested pump call processes no new requests and returns 0. Pending requests stay queued in each server's request_rx and are picked up on the next top-level pump tick, once the current handler unwinds its try-frame. The in-flight handler's own await still makes progress — its resolution drains through PENDING_RESOLUTIONS in js_stdlib_process_pending, not through this request loop.

The sibling pump js_http_process_pending (node:http) dispatches handlers via js_closure_call0/1 without a try-frame wrapper, so it does not accumulate try depth and is unaffected.

Validation

Minimal Fastify + @perryts/mysql server (two awaited queries per request), local MySQL:

Load Before After
500 reqs @ concurrency 30 crash (Try block nesting too deep) within the first few requests n/a
3000 reqs @ concurrency 50 crash all 202, server up, 3000 rows
5000 reqs @ concurrency 100 crash all 202, server up
50 sequential ok ok
  • cargo fmt --all -- --check: clean
  • cargo test --release -p perry-stdlib: 79 passed, 0 failed

js_fastify_process_pending dispatches each request handler inside a
setjmp try-frame (call_closure2_catching -> js_try_push). A handler that
awaits pumps the event loop (js_async_first_call ->
js_promise_run_microtasks -> js_stdlib_process_pending), which re-enters
js_fastify_process_pending while the outer handler's try-frame is still
open on the native stack. The re-entrant call started another request
handler, whose try-frame nested inside the suspended outer one. Under
concurrent load the nesting grew one level per in-flight request and blew
MAX_TRY_DEPTH (128), aborting with 'Try block nesting too deep' — observed
as a silent SIGSEGV (exit 139) under PM2 on Linux x86_64.

Add a thread-local re-entrancy guard: a nested pump call processes no new
requests and returns 0. Pending requests stay queued in each server's
request_rx and are picked up on the next top-level pump tick once the
current handler unwinds its try-frame. The in-flight handler's own await
still makes progress — its resolution drains through PENDING_RESOLUTIONS
in js_stdlib_process_pending, not through this request loop.

Repro (pre-fix): a Fastify + @perryts/mysql server crashes within the
first few requests under concurrent load (e.g. 500 reqs @ concurrency 30);
sequential load never crashes because requests never overlap. Post-fix:
5000 reqs @ concurrency 100 complete cleanly, server stays up.
@proggeramlug
proggeramlug merged commit ca66fff into main May 27, 2026
10 checks passed
@proggeramlug
proggeramlug deleted the worktree-fix-1824-fetch-mysql-segv branch May 27, 2026 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SIGSEGV: async frame slot mis-restored after await in a loop (js_object_get_field_by_name on garbage ptr; not GC)

1 participant