SIGSEGV handling POST requests in compiled Fastify + @perryts/mysql service (0.5.1026 / 46b80d78); GET unaffected

Summary

A Perry-compiled Fastify service (chirp, a small telemetry API) using @perryts/mysql SIGSEGVs deterministically when handling a POST request whose handler reads the JSON body and runs MySQL queries. GET/read endpoints are completely unaffected — the same process serves 8000 concurrent /health requests (each doing SELECT 1) with 0 crashes, but a single sequential POST to the ingest route crashes the process (signal 11 / SIGSEGV, core-dump), and under systemd it then crash-loops.

This is a regression: an older Perry build (~2026-03-14) of the same service handles these exact POST bodies in production every day. It first appears on perry 0.5.1026 (commit 46b80d78).

Environment

perry --version → 0.5.1026 (built from commit 46b80d78, v0.5.1025-130-g46b80d78)
Target: Linux x86-64 (Ubuntu 24.04), glibc 2.39
fastify (Perry-bundled), @perryts/mysql 0.1.4
Reproduces with both build profiles — identical backtrace:
- default (auto-optimize: trimmed stdlib, panic=abort)
- --no-auto-optimize (prebuilt full stdlib, unwind runtime)

Symptom / how to reproduce

The service is a standard Fastify app: a global onRequest CORS hook, route plugins registered with app.register(plugin, { prefix }), and a POST /api/v1/event handler that does roughly:

app.post('/event', async (request, reply) => {
  const rawBody = JSON.stringify(request.body);          // for HMAC
  const apiKey = request.headers['x-chirp-key'] as string;
  const auth = await query('SELECT id, hmacSecret FROM projects WHERE apiKey = ?', [apiKey]);  // @perryts/mysql
  // ...rate-limit (in-memory), validate, normalize dims...
  await query('INSERT INTO events (...) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)', [/* 11 params */]);
  reply.status(202);
  return { accepted: true };
});

Steps:

perry compile src/main.ts -o chirp (0.5.1026)
run it against a MySQL/MariaDB
curl -XPOST localhost:PORT/api/v1/event -H 'Content-Type: application/json' -H 'X-Chirp-Key: <key>' -d '{"event":"compile","dims":{"platform":"linux","status":"success"}}'
→ server SIGSEGVs (first or within a few sequential POSTs). GET /health before/after the POST is fine.

I can share the full failing source/binary on request.

What I ruled out (minimization matrix)

I tried to reduce this to a one-file repro. All of the following standalone Fastify programs survive thousands of POSTs (so none of these alone is the trigger):

Standalone repro	POST handler does	Result
1	read `request.body` only (no await)	survives
2	read body, then `await` a timer	survives
3	read body, then `await pool.query('SELECT 1')`	survives
4	`JSON.stringify(body)` + `await` a parameterized `INSERT`	survives
5	auth `SELECT ... WHERE apiKey=?` (via a `[rows,fields]` tuple adapter + extractRows) then parameterized `INSERT`	survives
6	read `body.dims` and probe 16 keys, most absent	survives

Yet the full app crashes deterministically. So the trigger appears to need the fuller app context (global onRequest hook + app.register-encapsulated route plugins + the full module graph), not any single operation above. Concurrency is not required (a single sequential POST crashes it); the connection pool is not the cause (concurrent reads are fine).

Backtrace

The compiled binary is stripped, and --debug-symbols is documented as a no-op on non-Windows targets, so I couldn't symbolize. The crash is on the main thread (Thread 1); all tokio workers are idle in epoll_wait/syscall. The backtrace is identical across runs and both build profiles:

Thread 1 "chirp" received signal SIGSEGV, Segmentation fault.
#0  0x...b9befb   #1 0x...3aab3   #2 0x...14e4a   #3 0x...48dc9
#4  0x...c2fcc    #5 0x...c7b35   #6 0x...e6e2e   #7 0x...74cf9   #8 0x...d3277
#9  0x...b943b8   #10 0x...d2baa  #11 0x...0eab7
#12 0x...b943b8   #13 0x...0e5f4  #14 0x...e8845
#15 0x...b943b8   #16 0x...e7e7f  #17 0x...448cb  #18 0x...452c2  #19 0x...c7b50

The repeated frame 0x...b943b8 at depths #9/#12/#15 looks like an async-resume/await trampoline appearing once per await in the handler chain.

Possible lead (uncertain)

An intermediate build (before I cleared a stale auto-optimize cache) emitted per-request debug output during POST body handling, from crates/perry-stdlib/src/fastify/context.rs json_value_to_jsvalue:

[req_json] body = {"event":"compile","dims":{"platform":"macos","target":"native","version":"0.5.1026","status":"success"}}
[json_value_to_jsvalue] Building object with 2 fields
[json_value_to_jsvalue] Setting field: event / dims / platform / target / version / status
[json_value_to_jsvalue] Post-build 'email' lookup: bits=0x7FFC000000000001 is_string=false is_undefined=true

The hardcoded 'email' field probe on a body that has no email field is suspicious (object field construction / field-name interning?). This may or may not be the crash site — flagging it as a lead.

Asks

Any pointers on what changed in fastify POST-body / request handling around 46b80d78 would help.
Could --debug-symbols (or an equivalent) retain symbols on Linux targets? Right now a SIGSEGV in a compiled Linux service is an unsymbolizable wall of ??, which makes runtime crashes very hard to report.

Happy to provide the full repro repo, the failing binary, or run further experiments.

Triage: this is almost certainly a manifestation of #1597 (Generational GC: selective evacuation segfaults async resumption — missing write-barrier/remembered-set edge), not a Fastify/@perryts/mysql-specific bug. The evidence lines up tightly:

Crash is on the main thread during async resumption. Your backtrace's repeated frame at Issue compiling a new project on Windows #9/String arguments corrupted when passed to cross-module functions inside callback scopes #12/Llvm backend #15 — "once per await in the handler chain" — is the async-resume trampoline. The POST handler awaits 3× (read body → auth SELECT → INSERT); GET//health awaits once (SELECT 1), which is why reads are fine and writes crash.
It needs the full app, not a one-file repro. Your minimization matrix (all 6 standalone POSTs survive) fits a GC bug: the crash needs enough live allocation + the right object graph (global onRequest hook + app.register plugins + full module graph) to (a) cross the nursery→old-gen evacuation threshold and (b) leave a live slot pointing at a forwarded object across an await. Standalone apps don't allocate enough to trigger policy evacuation.
It's a regression appearing at 0.5.1026. The C4b selective-evacuation policy (and its later δ tuning) landed in the 0.5.23x–0.5.10xx window — consistent with the ~2026-03-14 build being clean and 0.5.1026 crashing.

Immediate production workaround (no recompile)

The evacuation policy is gated by runtime env vars. To stop the crash-loop now, set on the running binary:

PERRY_GEN_GC_EVACUATE=0    # disables policy evacuation (keeps generational GC) — try this first

If it still crashes, fall back to:

PERRY_GEN_GC=0             # full mark-sweep, no generational/evacuation (slower but safe)

If PERRY_GEN_GC_EVACUATE=0 makes the crash disappear, that confirms #1597 (the bug is in the evacuate-and-rewrite path missing a live async-frame slot). PERRY_GC_VERIFY_EVACUATION=1 will panic at the exact offending slot if you can run a debug pass — useful for pinning the missing scanner.

Suggest consolidating this under #1597 (keep this open as the real-world repro / "flips to fixed when #1597 lands"). Happy to take the full failing source you offered — a deterministic repro would make pinning the missing remembered-set/write-barrier edge much faster.

fix(runtime): #1663 — re-entrant microtask drain nulls Task::Promise …

Thanks for the fix in b199d30c (#1675) — but it does not resolve this crash. Re-tested at exactly that commit and the service still SIGSEGVs on the first POST.

Toolchain (confirmed at the fix commit):

perry --version → 0.5.1026, git rev-parse HEAD → b199d30c
full cargo build --release (so libperry_runtime.a was rebuilt with the fix)

Recompiled the app three ways to rule out any stale Perry cache (the per-module object cache and the target/perry-auto-* trimmed libs both persist across Perry rebuilds and bit me earlier):

build	caches	result
default (auto-optimize, trimmed libs)	reused	SIGSEGV
`--no-auto-optimize` (fresh full stdlib)	—	SIGSEGV
`--no-cache --no-auto-optimize` + `rm -rf target/perry-auto-*`	fully fresh	SIGSEGV

In all three: process starts, GET /health returns 200, then it crashes on the first sequential POST to the body+DB handler. Read-only load is still completely fine (8000 concurrent /health doing SELECT 1 → 0 crashes); only the multi-await POST path crashes, and a single request is enough (no concurrency needed).

Fresh backtrace (fully-fresh --no-cache --no-auto-optimize build @ b199d30c):

Thread 1 "chirp" received signal SIGSEGV, Segmentation fault.
#0  0x0000555555ba97eb
#1  0x000055555563d6e3   #2  0x0000555555617a7a   #3  0x000055555564b9f9
#4  0x00005555557caa75   #5  0x00005555557d18cf   #6  0x0000555555bf689e
#7  0x0000555555677929   #8  0x00005555556d5e77
#9  0x0000555555ba1ca8   #10 0x00005555556d57aa   #11 0x00005555557116b7
#12 0x0000555555ba1ca8   #13 0x00005555557111f4

Same signature as the original report: crash on the main thread, all tokio workers idle, and the repeated frame 0x…ba1ca8 at #9/#12 is the per-await async-resume trampoline. The handler's await chain is ~3 deep: read JSON body → SELECT … WHERE apiKey = ? → INSERT … (all via @perryts/mysql). The re-entrant-microtask-drain fix evidently doesn't cover this nesting.

Binary is stripped so frames are ?? — the request from the OP to retain symbols on Linux (--debug-symbols is a no-op there today) still stands; I'd gladly attach a symbolized trace if there's a way.

Reopening. I have the toolchain + a standing repro environment, so I can test the next iteration immediately.

Dug into the reopen. Short version: the re-entrant-drain SIGSEGV is fixed by b199d30 on both macOS arm64 and Linux x86-64 — I couldn't get it to crash on a correctly-built b199d30c, and I've got disassembly + repro evidence for why. I also fixed the unsymbolizable-Linux-crash problem you raised (#2), which is the thing that'll let us nail down whatever you're still seeing.

Why `b199d30` fixes it

The crash was the Task::Promise arm reloading the running promise from a clobbered TLS cell after the callback and dereferencing (*promise).async_id (offset 0x30) on NULL. b199d30 roots promise/next in a handle scope and reloads them via RuntimeHandle::get_raw_mut_ptr after the callback. The key question was whether that reload survives optimization — so I disassembled js_promise_run_microtasks on both targets, panic=abort (the default auto-optimize profile):

macOS arm64 and Linux x86-64: the reload is a real out-of-line call get_raw_mut_ptr that re-resolves the GC-root handle stack. So even if a re-entrant drain reallocates that stack mid-callback, the reload reads the current slot, not a stale address. No dangle, no NULL deref.

I then ran 9 repros of your handler's shape and they're all byte-identical to node --experimental-strip-types:

prior completed awaiting-call (≈ onRequest hook) + a non-transformed async handler closure doing 3 sequential awaiting-calls (read-body → SELECT → INSERT)
timer-based real-async awaits (not just Promise.resolve())
a Fastify-style dispatch graph (hook → handler via app.register-ish wrappers)
heavy allocation under forced GC evacuation (PERRY_GC_FORCE_EVACUATE=1 PERRY_GC_VERIFY_EVACUATION=1) to rule out the Generational GC: selective evacuation segfaults async resumption (missing write-barrier/remembered-set edge) #1597 evacuation angle — clean

So why are you still crashing at `b199d30`?

Most likely a stale runtime .a. The fix lives in libperry_runtime.a, which is linked into your compiled program, not into the perry binary. There are two cache layers that survive a plain rebuild and bit me too:

the per-module object cache (.perry-cache/objects/)
the auto-optimize trimmed libs (target/perry-auto-*/)

And a sharp edge: if your perry binary's path resolves (via symlink) into a different perry checkout, the auto-optimize step compiles the runtime from that checkout's source. A genuinely clean rebuild:

git -C <perry-repo> rev-parse HEAD          # confirm you're really at the fix
cargo build --release -p perry-runtime -p perry-stdlib -p perry
rm -rf target/perry-auto-*                  # in BOTH the perry repo and your app
rm -rf <app>/.perry-cache
<perry-repo>/target/release/perry --no-cache <app>/src/main.ts -o chirp

If it still crashes after that, it's something your full app/data hits that I can't synthesize — and now we can see it:

`--debug-symbols` now works on Linux/macOS (#1692)

It was a no-op off-Windows before. As of #1692 it retains the symbol table (js_* + your functions) and emits -g DWARF, so:

perry --debug-symbols src/main.ts -o chirp
lldb -k bt -o run ./chirp     # or: gdb -ex run -ex bt ./chirp

gives real frames + file:line instead of ??. A symbolized backtrace from your repro would let me pin the exact frame immediately — and your offer of the full failing source/binary still stands, I'll take it.

PR: #1692 (debug-symbols on Linux/macOS + a deeper re-entrant regression test + a small hardening so after() never dereferences a reloaded promise).

fix(#1663): --debug-symbols retains symbols on Linux/macOS + harden m…

fix(node:async_hooks): complete promise/await lifecycle + retain #1762 …

fix(runtime): Buffer.from() handles inline SSO short strings (#1767) (#…

fix(gc): root the microtask runner's callback and value across the ho…

fix(gc): Promise.all at scale read globalThis and its own combinator …

Navigation Menu

Repository navigation

SIGSEGV handling POST requests in compiled Fastify + @perryts/mysql service (0.5.1026 / 46b80d78); GET unaffected #1663

Summary

Environment

Symptom / how to reproduce

What I ruled out (minimization matrix)

Backtrace

Possible lead (uncertain)

Asks

Immediate production workaround (no recompile)

Why `b199d30` fixes it

So why are you still crashing at `b199d30`?

`--debug-symbols` now works on Linux/macOS (#1692)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SIGSEGV handling POST requests in compiled Fastify + @perryts/mysql service (0.5.1026 / 46b80d78); GET unaffected #1663

Description

Summary

Environment

Symptom / how to reproduce

What I ruled out (minimization matrix)

Backtrace

Possible lead (uncertain)

Asks

Activity

proggeramlug commented on May 24, 2026

Immediate production workaround (no recompile)

proggeramlug commented on May 24, 2026

proggeramlug commented on May 24, 2026

Why b199d30 fixes it

So why are you still crashing at b199d30?

--debug-symbols now works on Linux/macOS (#1692)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Why `b199d30` fixes it

So why are you still crashing at `b199d30`?

`--debug-symbols` now works on Linux/macOS (#1692)