Repository navigation
Native shop-admin signup SIGBUS in toUser → dtRequiredToIso (downstream of @perryts/mysql MyDateTime.toDate corruption) #859
Description
Symptom
Native-compiled skelpo-shop-admin (./build/server) at perry 0.5.914 crashes with SIGBUS / exit 138 during POST /v1/auth/signup. The first INSERT INTO users commits (visible in MySQL), then the process dies — curl gets Empty reply from server and the connection drops. tsx server/main.ts against the same source + same DB returns the correct JSON / HTTP 201.
This is the v0.5.914 surfacing of what at v0.5.912 was a silent body-0/HTTP 200 — same site, harder failure now that the Invalid-Date sentinel is in.
Pinned crash site (checkpoint-instrumented trace)
Instrumented server/persistence/users.ts::createUser/getUserById/toUser:
U1 INSERT ← createUser INSERT runs
U2 done, pre getUserById ← INSERT resolved, row visible
G1 pre queryOne ← getUserById entered
G2 queryOne obj ← SELECT returned the row
T0 enter (toUser entered)
T1 id=019e2fac-... ← bufToId(r.id) works
T2 emailVerified=null ← dtToIso(null) works
T3 totp=null ← dtToIso(null) works
← SIGBUS. No T4.
Next statement is dtRequiredToIso(r.createdAt), which calls r.createdAt.toDate().getTime() on the MyDateTime object the @perryts/mysql driver decoded. That .toDate() call is where the process dies.
Root cause
The proximate trigger is #788 — closure-captured numeric params read as 0 inside object-literal : Date method (filed separately). MyDateTime.toDate() is exactly that shape, so for every decoded DATETIME the captured y/mo/d/... read as 0 inside the method and new Date(Date.UTC(0, ...)) produces a year-0 Date (-62155492224598 ms). When that value flows into dtRequiredToIso → d.toISOString() and the resulting string into perry's JSON / fastify reply path, something in the formatting / serialization chain assumes a finite, plausible timestamp and segfaults.
In a standalone repro of just the MyDateTime.toDate().toISOString() shape (no Fastify, no mysql connection — see #788) the same code returns a garbage 0000-05-15T17:29:36.598Z string without crashing. The shop-admin path additionally goes through fastify's JSON reply, the post-#748 wait_for_promise condvar path, and likely the @perryts/mysql worker-thread → main-thread Date hand-off. One of those layers can't survive a year-0 / Invalid Date and faults.
Repro
Local: clean checkout of skelpo-shop-admin, cargo build --release of perry at HEAD (e18cdb07 / v0.5.914), then:
mysql -uroot -e "DROP DATABASE IF EXISTS shopadmin; CREATE DATABASE shopadmin"
KEK_BASE64=... JWT_SIGNING_KEY_PEM=... DB_DSN='mysql://root:@127.0.0.1:3306/shopadmin' \
BIND_ADDR='127.0.0.1:18080' ./build/server &
curl -X POST http://127.0.0.1:18080/v1/auth/signup \
-H 'content-type: application/json' \
-d '{"email":"x@y.com","password":"correct-horse-battery-staple","accountName":"X"}'
# → curl: (52) Empty reply from server; server process exits 138.
tsx server/main.ts against the same DB+env returns the correct {accessToken, refreshToken, expiresAt, user, account} payload with HTTP 201 and all five tables populated.
Tracking history
This is the same end-user failure that drove #665 and #748:
- post-#652: CJS module.exports = Class + extends still drops methods (rate-limiter-flexible repro) #665 — compilePackages-overrides-native-modules + cross-class method dispatch (closed, fix verified at v0.5.899).
- Native route returns body
0+ HTTP 200 after sequence of @perryts/mysql writes; first INSERT commits, rest silently no-op, explicit return dropped #748 —wait_for_promise1s polling cap (closed at v0.5.912), then Invalid-Date sentinel (closed at v0.5.914 bydc6be087). Both landed correctly. - This issue — the remaining native-server blocker, fully explained by AsyncLocalStorage: real async-context tracking across await / microtasks / timers #788 (closure captures in
: Dateobject-method) once that's fixed. Fixing AsyncLocalStorage: real async-context tracking across await / microtasks / timers #788 should make this go away; filing separately so the segfault stays visible in the meantime in case the downstream formatter/reply path needs a defensive fix even after AsyncLocalStorage: real async-context tracking across await / microtasks / timers #788 lands.
Suggested fix scope
- Fix AsyncLocalStorage: real async-context tracking across await / microtasks / timers #788 (the captures) — that should be enough for the user's signup to work end-to-end. The corrupt-Date path stops producing year-0 values and
toISOStringreturns the real timestamp. - Independently, find the downstream layer that segfaults on a year-0/Invalid Date and harden it — same value at the same call site shouldn't take down the process. (My standalone repro of the same shape returns a garbage string without crashing, so the segfault is specific to one of: fastify reply path, @perryts/mysql worker→main thread Date crossing, or
wait_for_promiseresuming with a Date argument it doesn't expect.)
Perry version at repro: 0.5.914.
Activity
proggeramlug commentedon May 16, 2026
Reopening — the #858 fix landed correctly (verified at v0.5.922; all my prior standalone Date repros pass byte-identical to tsx, and
MyDateTime.toDate()now returns valid Dates with valid millis). That removed the proximate trigger I'd named, but the shop-admin signup still SIGBUSes onPOST /v1/auth/signup— same exit-138 symptom as before, different position.What changed (post-#858)
With heavy persistence-layer instrumentation (U/G/T/S checkpoints in
users.ts,pool.ts,auth.ts), the trace runs much further than before:The #858 fix made the original pinned site (
dtRequiredToIso(r.createdAt)insidetoUser) work end-to-end. Crash has moved to betweenS9 pre createAccountand thecreateAccountfunction body entering — i.e. inside the expressionawait createAccount({ name: …, trialEndsAt: trialEndsAt() }).And it's instrumentation-sensitive
Same source, same perry 0.5.922 build, only the volume of
console.errorcalls differs:S9 pre createAccount(dies insideawait createAccount(...))console.errorin auth route)ZZ3 pre-createUser(dies insideawait createUser(...))That's the diagnostic — the SIGBUS position is not a fixed codegen site; it shifts with allocation/GC pressure or microtask-queue depth. Same code, same DB, same env, just fewer/more
console.errorcalls on the path before the failure point. Strongly suggests a memory-pressure / GC-during-await / async-chain-state interaction rather than a single mis-compiled statement.Concrete repro
Local: clean checkout of skelpo-shop-admin, perry at HEAD (v0.5.922):
tsx server/main.tsagainst the same DB+env still returns the full{accessToken, refreshToken, expiresAt, user, account}JSON with HTTP 201 and all five tables populated.What I'd suggest next
The position-shifts-with-instrumentation behavior is the most actionable clue. Suspects, in order:
users.ts/pool.tsallocate strings (UUID hex, MySQL date format strings, JSON bodies) —console.errorcalls also allocate. Different allocation count → GC fires at a different point → different live promise/captured slot freed.For (1) I'd capture the SIGBUS via
RUST_BACKTRACE=full+lldbon the compiled./build/serverto get the faulting PC and the LLVM-IR__initsymbol it's in. I triedRUST_BACKTRACE=fullalready and nothing was emitted before the process died — likely needs a SIGBUS handler installed inperry-runtime(or a wrapper script catching the crash and runninglldb -pon the core). Happy to do that if you can point at a recipe perry uses internally for this kind of in-the-field crash capture.Perry version at repro: 0.5.922 (HEAD
4edcb077). Reopening since the symptom persists and the position-shift makes it look like a runtime/codegen interaction rather than a single missing fix; the original #859 title is no longer accurate (toUserworks fine), so a rename to something like "native shop-admin signup SIGBUS during multi-await Fastify handler; crash site shifts with instrumentation volume" would track it better.proggeramlug commentedon May 16, 2026
v0.5.923 with
e9186c7c(pin native-bridged Promises while in flight on tokio worker) is partial progress — but signup still SIGSEGVs / SIGBUSes deterministically, just at a moving site.Concrete evidence: the fix moved the needle but didn't close it
Two full-rebuild runs at perry 0.5.923, same shop-admin source, same DB, same env. Only the volume/placement of
console.errorcheckpoints differs.Run A — heavy checkpoints in
users.ts(T0–T6insidetoUser) +auth.ts(S7–S15):DB after Run A:
accountsnow has 1 row — pre-fix it was always 0. SocreateAccount's INSERT promise is surviving long enough to commit. Crash is post-INSERT, before the route handler'sawait createAccount(...)resumes (no S10).Run B — heavy checkpoints in
accounts.ts(CA1–CA6,TA0–TA5) + sameS7/S8/S9/S10inauth.ts, minusT*intoUser:DB:
Identical perry, identical user code (the only diff is which functions I added
console.errorcalls to). Crash position jumps from "insidecreateAccountawait-resume" all the way back to "insidecreateUser'sawait getUserById".accounts=1vsaccounts=0is the clean tell that the work did execute in Run A, just unwound badly.What this means for the fix
e9186c7ccorrectly pins promises atspawn_for_promise/spawn_for_promise_deferred/ the third entry point — and the heavy-instrumentation run proves the pinning is doing real work (thecreateAccountINSERT promise survives to commit, where pre-fix it didn't). But there's clearly at least one more code path where a Promise's pointer crosses to a worker without being rooted; instrumentation that perturbs allocation timing changes which path's window the GC happens to land in.Candidates worth checking, based on the trace + the fix's own framing:
getXById's SELECT promise after the INSERT increateUser/createAccount— Run B dies onawait getUserById(id, conn)insidecreateUser's body (U2 prints, U3 doesn't). That promise also goes main → @perryts/mysql worker → main; check whether its allocation site goes through one of the three patched entry points or a fourth.async function f() {...}allocatesP = js_promise_new()to return; iffthenawaits a native promise on a worker,Pis the "outer" promise that sits in the parent's await loop. Its lifetime crosses worker boundaries indirectly. Pinning the bridged promise alone doesn't necessarily root the awaiting outer promise..then(continuation)microtask, thecontinuationclosure captures the outer async state machine; if that capture's holder is an unrooted slot, GC can still sweep it before the microtask runs.The position-sensitivity to
console.errorcount is the diagnostic — it says the freed-pointer window is small and adjacent allocations decide whether the swept page has been reused/unmapped yet.Repro
Local: clean shop-admin checkout, perry HEAD (
v0.5.923, fullcargo build --release), MySQL up with emptyshopadminDB. SinglePOST /v1/auth/signup→curl: (52) Empty reply from server, server exits 138/139 (varies). My instrumentation diffs are in/tmp/users.bak9//tmp/accounts.bakBetc. on this machine if useful — happy to commit them as atest-files/test_issue_859_signup_chain.tsshape if you want a deterministic-enough in-tree repro (the catch being it needs a live MySQL on127.0.0.1:3306, which I think is why the prior agent could only do the synthetic Promise-tuple repro).For the next pass,
MallocStackLogging=1on macOS +lldb -ponce it crashes would show the freed-then-reused promise allocation's lastjs_promise_newcallsite — that should point at the unrooted entry path #4. Happy to drive that locally if you can sketch the right perry-side hook.Reopening; suggested rename: "native shop-admin signup still SIGSEGVs after #873 — additional Promise→worker entry path not pinned (position varies with instrumentation)".
proggeramlug commentedon May 17, 2026
New diagnostic data at v0.5.934 — the crash is not GC-related. The pinning fix in
e9186c7cis doing real work (more code paths complete than before) but it's chasing the wrong root cause for the residual signup crash.(1) Captured an
lldbbacktracelldb -batch ./build/server, triggered signup with one curl, caught the fault:The faulting instruction is
ldr w26, [x19, #0x4]— a 4-byte load at offset 4 of whatever's inx19.x19holds0x726f727249. That's ASCII bytes49 72 72 6F 72= "Irror" (little-endian as the CPU read them off memory). A 5-byte string fragment, not a heap pointer. A real Promise/object pointer on macOS arm64 would be in the0x000000010xxxxxxx(binary) or0x000000014xxxxxxx(mmap'd heap) range —0x726f727249is neither.This is type confusion — somewhere a value that is a string (or contains string bytes in low-order positions) is being treated as an object pointer and
[ptr + 0x4]is dereferenced. Not a use-after-free where a real but freed pointer is dereferenced — those would fault at an address that looks like a heap address, possibly returning garbage instead of crashing on a clearly-invalid one.(2) PERRY_GC_DIAG=1 — zero GC events during signup
Ran with
PERRY_GC_DIAG=1; the failing signup produces zero GC-related lines between server-up and crash. So the pinning code path the fix protects never even fires during the failing window — GC isn't running here.(3)
PERRY_GC_DISABLE=1bisect — same crashTo rule it out conclusively I patched a
PERRY_GC_DISABLE=1knob intogc_collect_inner()(no-op every collection, leak everything), rebuiltperry-runtime, ran signup. Same SIGSEGV exit 139, same address shape, sameusers=1 accounts=0DB signature as without the knob. (Knob reverted; checkout clean.)So: GC fully disabled → same crash → the residual #859 bug is not a GC-rooting issue. The pinning fix in
e9186c7cis correct for theusize-as-Promise-pointer-across-tokio path it patches, and the heavy-instrumentation run that producesaccounts=1proves it's doing real work for some other request shape. But the signup-failure path is hitting a different bug, in a different code site, with a different root cause.(4) Where to look instead
Type confusion at
[x19+0x4]reading a "Irror" string fragment — the question is which code site is calling a method on a value it expects to be an object but that's actually carrying string bytes in the low 48 bits. Two hypotheses:a. NaN-box tag-strip mistake. A NaN-boxed string
0x7FFF_726F_7272_49xxwhose top 16 bits get zeroed by a wrong mask in someunbox_to_pointerpath. Hits whenever the bottom 48 bits happen to look like a not-immediately-faulting address (this one isn't real but a malformed mask could still produce an address that's deep enough into a string buffer to read past it and fault on the next page).b. A
Result::Err(reason)from a stdlib bridge whose reason-pointer is being passed where the resolve-value pointer was expected.error/Errorstrings show up all over the binary (rustls error variants, mysql2 error paths,RateLimiterQueueError). If a stdlib reject path stores the error message's first 5–8 bytes into a slot that the resumer reads as*Promise, you'd get exactly this —EXC_BAD_ACCESSat a small-looking address that decodes to ASCII letters.The heavy-instrumentation run reaches
S9 pre createAccountandaccounts=1(the createAccount INSERT committed) — so the failing await is most likely the route handler'sawait createAccount(...)resumption, reading the resolved Account value to bind it toconst account. That's the value that gets dereferenced at[x19+4]. IfcreateAccount's returned Account pointer got swapped with an error-string pointer somewhere on the resolve path, you'd hit this exact shape.(5) What the failure mode looks like for tracking
0x726f727249decodes to "Irror" — not a heap pointer.console.errorcount → register/spill layout decides which of multiple sites the bad value reaches the deref. Suggests a single bug producing a small bag of corruptible values, with the spill landing on different code paths.70868f9a/ v0.5.934, perry-runtime + perry-stdlib rebuilt, all instrumentation reverted, no half-fixes.Happy to drive a
samply/instrumentscapture or build perry with debug symbols if that'd help —nm build/serveris fully stripped so my frame symbolication came back as__mh_execute_header + offset, which the maintainer's local symbol cache (or acargo buildwithout--release) would resolve immediately.proggeramlug commentedon May 17, 2026
Two new artifacts that should unblock the maintainer-side investigation:
(1) Standalone 50-line repro — no shop-admin source needed
issue-859-repro/repro.tsin the user repo (will share separately if useful). The full ingredient list still needs to be present to reproduce —@perryts/mysql+ argon2 + rate-limiter + Fastify route + theawait getById(id)afterawait exec(INSERT)pattern with theMyDateTime.toDate()consumer — but it's now decoupled from the shop-admin codebase. Same SIGSEGV exit 139, sameusers=1 accounts=0DB signature, same "Error" faulting address.Setup is two tables + a 50-line
.ts; full instructions in the includedREADME.md. Should be the easiest possible reproduction without the full shop-admin tree.(2) Sharper crash address — "Error" not "Irror"
Re-read x19 with debuginfo build:
x19 = 0x000000726f727245. I'd off-by-one'd a byte earlier; the actual bytes are45 72 72 6F 72= ASCII"Error"(capital E), not "Irror". That's the string"Error"being dereferenced as a Promise/object pointer. The instruction isldr w26, [x19, #0x4](load 32-bit at offset 4 — likely a class-id, promise-state, or header field load).x21/x22both hold0x0000020000341b08— a real heap pointer (probably the correct receiver of the method call, surviving in another register). That's a tell that the bug isn't "the whole object went bad"; it's "one specific slot got swapped with a string value."Two specific hypotheses to check first
a. Rejected-promise reason in resolve slot. If somewhere a
js_promise_resolve(p, value)is being called withvalue = errorString(the rejection reason) instead ofjs_promise_reject(p, reason), the awaiter unwraps it as a resolved value, then dispatches.something()on it. The receiver is"Error"bytes;[receiver+4]faults. The "Error" string is exactly what an error message would start with (new Error("...")produces"Error: ...").b. NaN-box mask error on a small/inline string. A
unbox_to_i64path that strips the high 16 bits of a NaN-box without checking the tag would convert a small-string-encoded JSValue into its raw byte payload as a pointer. If perry has any inline-string optimization for short strings (≤5 bytes — "Error" is 5), this hits.Symbol resolution
Still stripped —
perry compile's link step-dead_strips the DWARF references even whenperry-runtime/perry-stdlibare built withRUSTFLAGS="-C debuginfo=2". The maintainer's local build with full debuginfo + animage lookup --addresson the 6 frames in the README should resolve the function names instantly. I confirmed the addresses are stable across reruns (same instruction at+2212of the same outer function each time).Status
v0.5.936(HEAD70868f9a+ 1 build of debug-info perry-runtime/stdlib, no source edits).issue-859-repro/repro.ts+README.mdready to copy into perry'stest-files/if useful.Let me know if it'd help to (a) run
samply/instrumentsover the repro for a flamegraph of the failing call path, or (b) try to removeargon2/rate-limiterindividually from the repro and see which one is load-bearing. I tried a quick variant matrix but the shell driver had a bug — happy to redo it cleanly.proggeramlug commentedon May 17, 2026
MySQL-free 15-line repro
Bisected the failing signup down from
~250 LOC + 5 native deps + 2 tables + argon2 + rate-limiter + dtToIso + DATETIME columns + Date params + @perryts/mysqlto this:Same exact crash signature as the full shop-admin signup:
3/3 runs reliably reproduce.
tsx minimal.tsreturns the correct JSON HTTP 201.Load-bearing ingredients (drop any → no crash)
Found by exhaustive bisect (numbered as v1–v35 against my local checkout):
const toUser = …withfunction toUser(…) {…}((r)=>null)(undefined)await new Promise(r => setTimeout(r,1))instead ofawait delay()await Promise.resolve()(no named async fn)awaitinsidecreateUser, keep arrow callcreateUser, inline its body in the route handlermain()@perryts/mysql+ argon2 + rate-limiter + dtToIso + DATETIME cols + Date paramsSo the exact minimum trigger is:
async function(not an inlineawait new Promise(…)and not anawait Promise.resolve())const arrow = (x) => …(not afunctiondeclaration, not an IIFE)That's it. No mysql, no argon2, no rate-limiter, no Dates. Just the resume-from-an-awaited-named-async-fn followed by calling-a-module-level-arrow inside a Fastify-callback-invoked async frame.
Why this matches the "Error" address
x19carries the bytes45 72 72 6F 72= ASCII"Error". That has to be coming from somewhere in either:fastifypackage source (rustls / hyper error variants are full ofErrorstrings)new Promise(...)constructor reject-path stringWhatever it is, it's getting loaded into
x19where the codegen-emitted async-resume + arrow-call dispatch expects a Promise/object pointer — almost certainly an arrow-const's receiver slot or the resumed async-frame's return-value slot getting overwritten with a string reason from a prior reject path. The arrow-vs-function distinction + the "named-async-fn-only" distinction are the strong tells; both touch perry's closure / async-state-machine codegen.Files
issue-859-repro/minimal.ts(the 15-liner above) andMINIMAL.md(fullcontext + bisect table) checked in to the user repo. No external setup
needed — just
fastifyinnode_modules. Happy to PR it into perry'stest-files/directory if useful.Local checkout still clean at
v0.5.936, no source edits, full instrumentation reverted.fix(codegen): #859 — module-level const arrow callee inside resumed a…
fix(codegen,hir): #859 — module-level const arrow callee inside resum…
fix(codegen,hir): #859 — module-level const arrow callee inside resum…
proggeramlug commentedon May 17, 2026
Verified at v0.5.953.
(a) The 15-line minimal MySQL-free repro now passes:
POST /go→{"ok":true,"u":null}HTTP 201, server stays alive. The fix in0ef026f7("module-level const arrow callee inside resumed async-step body") correctly closes the exact shape I'd bisected to.(b) The full shop-admin signup still SIGSEGVs, but at a different site with a different signature — significant additional ground gained, one more bug to chase.
What the fix unblocked
DB state after one signup at v0.5.953 (vs pre-fix):
usersaccountssessionsmembersauditLogcreateAccount,addMember,issueSessionall complete and commit now. Trace via 15 inlineconsole.errorcheckpoints in the signup route runs cleanly throughZ13 session done. The next line is the synchronousconst accessToken = issueAccessToken({...})— crash there.New crash signature
x0 (the receiver of
[x0+4]) holds0xe6 = 230— small integer (class ID? property index? count?) being dereferenced as a pointer. Not the same "Error"-string type-confusion as the #859 fix targeted. Different instruction context, different register state, different fault address shape.Standalone narrows
issueAccessTokenis one line:Pulled into a minimal Fastify route with the same shape (
async asyncStep→jwt.sign(claims, PEM, {ES256,...})) — passes byte-identical to tsx. So the bug is not injsonwebtoken.signitself; it requires the accumulated state from the 6 prior awaits (rate-limiter, getUserByEmail, hashPassword, createUser, createAccount, addMember, issueSession) for the values reachingjwt.signto carry whatever corruption triggers the deref.The most suspect values reaching
issueAccessToken:user.id— string returned through the 4-level chaincreateUser → getUserById → toUser → bufToId(buf)(thebufToIdarrow-const that worked after the Native shop-admin signup SIGBUS in toUser → dtRequiredToIso (downstream of @perryts/mysql MyDateTime.toDate corruption) #859 fix)account.id— same shape"owner"string literal,nullPossible the same arrow-via-resumed-async-step shape that #859 fixed has a sibling case for the return values flowing through it (not just the call). Or a separate codegen issue at the
function … { return jwt.sign(...); }site combined with the multi-await preamble.Status
#859itself — the bug it was opened for — is closed-and-verified. The remaining shop-admin signup blocker is a distinct issue. Local checkout atv0.5.953, no source edits, all instrumentation reverted. Filing the new crash separately so the tracking stays clean; will link from this thread once filed.Happy to bisect the new site toward a MySQL-free minimal the same way I did for #859 — should be straightforward since I know the failure transition is
Z13 (issueSession resolved) → issueAccessToken(...). Should I drive that locally and post the minimal repro to the new issue?