SIGSEGV: async frame slot mis-restored after await in a loop (js_object_get_field_by_name on garbage ptr; not GC)

Summary

A compiled Fastify + @perryts/mysql service (gscmaster-api) SIGSEGVs (exit 139, no JS error, PM2 restart) during a long-running background job that performs a large number of sequential await fetch() (HTTPS) interleaved with await mysql execute() in a single async invocation.

This looked like #1597 at first, but it is not that generational-GC bug:

Rebuilt the compiler from current main (includes fix(gc): barrier Promise pointer stores for async resumption #1715 fix(gc): barrier Promise pointer stores) → still crashes.
PERRY_GEN_GC_EVACUATE=0 → still crashes.
PERRY_GEN_GC=0 (full mark-sweep, generational GC fully disabled) → still crashes.

So the GC fix and both GC-disable workarounds don't help — this is a distinct failure.

Shape of the workload

The job processes three "date ranges". For each range it does, for ~37 items, sequentially:

2× await fetchPostWithAuth(googleapis.com, ...) (stats)
1× await fetchPostWithAuth(...) (daily)
4× await fetchPostWithAuth(...) (dimensions)

i.e. ~7 awaited HTTPS round-trips per item × 37 ≈ ~260 sequential await fetch() per range, each followed by await execute("INSERT ... ON DUPLICATE KEY UPDATE", ...) writing JSON blobs up to ~15 KB via @perryts/mysql.

Observed boundary: one full range completes and persists all its DB writes; the process crashes near the start of the second range (the next fetch), ~40–43 s in. One range ≈ ~260 fetches + ~230 inserts = OK; two ranges reliably crash. The crash is silent (empty stderr at the crash point; PM2 sees a restart). Reducing the work to a single range avoids it.

This points at a resource that accumulates across many sequential awaited fetch/DB calls and isn't released (sockets/connections, async frames, or similar) rather than a single offending statement — the same token/site/code path works fine for the first ~260 calls and dies on the ~261st.

Environment

Perry: PerryTS/perry main @ c720d2a4 (rebuilt 2026-05-25; includes fix(gc): barrier Promise pointer stores for async resumption #1715).
Compiled Fastify service (plain Fastify), @perryts/mysql (mysql2-compatible) pool connectionLimit: 10, await pool.execute(...).
fetch via the built-in fetchWithAuth / fetchPostWithAuth helpers.
Linux x86_64, run under PM2.

Asks

Is there a known cap / leak on the number of sequential fetch() (or awaited mysql execute()) in one async invocation? Any PERRY_* knob to test (e.g. fetch connection pooling)?
Happy to produce a trimmed standalone repro (loop of N await fetch() to a local server + N inserts) and bisect N — say the word and I'll attach it plus a core dump / PERRY_* trace if useful.

(Originally mis-attributed to #1597 — corrected: that fix and PERRY_GEN_GC=0 both leave this crash in place, so filing separately.)

Update: rebuilt the compiler from main @ 4a07ba07 (includes #1825 fix(gc): root IMPLICIT_THIS …, which closed #1813) and redeployed — this crash persists. Same signature: one date-range's worth of work (~260 sequential await fetch() + @perryts/mysql execute()) completes and its DB writes land, then SIGSEGV (exit 139, PM2 restart) ~50s in, near the start of the second range's first fetch. So #1825 addresses a different receiver-rooting case; it doesn't cover this one.

Recap of what does not help: #1715, #1825, PERRY_GEN_GC_EVACUATE=0, PERRY_GEN_GC=0 (full mark-sweep). Still happy to put together a trimmed standalone repro (loop of N await fetch() to a local server + N inserts, bisecting N) — just let me know if that'd help.

fix(gc): restore shadow stack on exception unwind (#1830) (#1837)

fix(fastify): guard request pump against re-entry (#1824) (#1932)

Update: symbolized backtrace — async frame slot mis-restored after an `await` in a loop

I got a gdb backtrace on a PERRY_DEBUG_SYMBOLS=1 build (attach to the live process, catch SIGSEGV). This refines the diagnosis substantially and rules out my earlier guesses. This is not GC, not rooting, not the worker-thread-allocation issue (that turned out to be a separate real bug — see #1957 — which does not fix this crash).

Faulting backtrace

Thread 1 "…" received signal SIGSEGV, Segmentation fault.
#0  js_object_get_field_by_name ()
#1  js_object_get_field_by_name_f64 ()
#2  perry_closure_lib_<module>_ts ()                       ← resumed continuation closure
#3  perry_runtime::promise::async_step::async_step_fulfill_thunk ()
#4  js_promise_run_microtasks ()
#5  perry_fn_lib_<module>_ts__buildSummaryCache ()         ← the async fn
#6  perry_closure_lib_<module>_ts ()
#7  js_promise_run_microtasks ()
#8  perry_stdlib::fastify::server::wait_for_promise ()
#9  perry_stdlib::fastify::server::js_fastify_process_pending ()
#10 js_stdlib_process_pending ()
#11 js_run_stdlib_pump ()
#12 main ()

rip = js_object_get_field_by_name+217
rdi = 0x200000000        ← the OBJECT argument: garbage (not a heap address)
rsi = 0x2f90ce18900      ← the field-name string: valid heap ptr

The fault address is rdi - 8 (0x1fffffff8) — js_object_get_field_by_name reading the GC header of the "object". rdi (the object) is 0x200000000, which is not a valid heap pointer (real heap objects in the same run are at 0x2f90…). It also varies run-to-run (seen 0x200000000 and 0x1fffffff8), i.e. it's an uninitialized / mis-restored frame slot, not a freed-then-reused pointer (a freed pointer would still be a heap address).

What the code is doing

It's an async fn that does await inside a loop and maintains a long-lived heap accumulator across the awaits (simplified from the real buildSummaryCache):

async function buildSummaryCache(...) {
  const dDates: string[] = [];
  const dAgg: any[] = [];                         // long-lived, captured across every await
  for (let i = 0; i < sites.length; i++) {
    const dailyStr = await getCacheEntry(...);    // FFI-async await (a @perryts/mysql query)
    const dailyArr = JSON.parse(dailyStr);
    for (let j = 0; j < dailyArr.length; j++) {
      const row = dailyArr[j];
      // ... find idx in dDates ...
      const existing = dAgg[idx];
      existing.clicks = existing.clicks + (row.clicks || 0.0);   // ← faults: existing/row is the garbage ptr
      // else: dAgg.push({...})
    }
  }
}

The crashing access is the numeric field read (js_object_get_field_by_name_f64) on a frame-captured object after the resume. So a frame local (the accumulator element existing, or row/dailyArr) is mis-restored to garbage when the continuation resumes after await getCacheEntry.

Confirmed NOT the cause (all tried, crash unchanged)

PERRY_GEN_GC=0, PERRY_GEN_GC_EVACUATE=0, PERRY_GC_FORCE_EVACUATE=1+PERRY_GC_VERIFY_EVACUATION=1
PERRY_GC_DIAG=1 shows no GC fires before the crash — so it's not a collection freeing/moving the frame.
The Generational GC: selective evacuation segfaults async resumption (missing write-barrier/remembered-set edge) #1597 (fix(gc): barrier Promise pointer stores for async resumption #1715) and SIGSEGV race in js_native_call_method via @perryts/mysql nativeScramble during concurrent Pool.acquire handshakes (0.5.1029 / c720d2a4) #1813 (fix(gc): root IMPLICIT_THIS so moving GC can't strand the dispatch receiver (#1813) #1825) GC fixes.
A promise-reaction-rooting experiment (rooting the in-flight native promise's on_fulfilled chain).
The fix(mysql2): build query result on the main thread, not the tokio blocking worker (worker-thread JSValue construction is UB) #1957 mysql2 worker-thread-JSValue fix (gdb-confirmed identical crash with it applied).

Reproduction notes

Reproduces deterministically in a compiled Fastify + @perryts/mysql service: a background job runs ~3 "passes", each ~37 iterations of the loop above (each pass does a few hundred awaited mysql queries + writes). It completes the first pass and crashes early in the second (~50s in, ~the 260th awaited op).
A minimal repro of just the pattern (await fetch() in a loop + the same accumulator, 3×37 iterations) completes cleanly — so the trigger needs the larger cumulative workload (heap state / number of awaited ops), not just the shape. I haven't isolated the minimal trigger yet.

Offer

I can provide: the symbolized binary + a core dump, disassemble js_object_get_field_by_name / the resumed closure's call site, or help bisect a minimal repro (e.g. growing the iteration count / using @perryts/mysql specifically vs fetch). Happy to test patches against the deterministic service repro. What would be most useful?

(Separately, the worker-thread JSValue construction in perry-ext-mysql2 that I found while investigating this is fixed in #1957.)

Found and fixed a concrete frame-loss bug in this symptom class — PR #1965

While chasing this I found a deterministic bug that produces exactly the failure you describe (a frame slot live across an await inside a loop getting lost on resume → garbage-pointer deref / SIGSEGV when the slot is an object, wrong value when it's a number).

Root cause: the async→generator transform splits an async body into a state machine at each await. Its body walkers were written for for / while / if / try / switch and never handled do { … } while or labeled loops (outer: for (…)). So an await inside one of those was mishandled three ways: loop-body const/lets weren't hoisted/boxed (lost across the await), the buried yield wasn't detected, and linearize_body had no arm for them so the whole loop was emitted unsplit (the await never became a resume state).

Minimal repro (was wrong / could crash; correct after the fix):

async function ident(x: number) { return x; }
async function f(n: number) {
  const agg: { clicks: number }[] = [];
  outer: for (let i = 0; i < n; i++) {
    const row = { clicks: i * 5 };     // object created before the await
    const got = await ident(i);
    if (agg.length === 0) agg.push({ clicks: row.clicks + got });
    else { const e = agg[0]; e.clicks = e.clicks + row.clicks + got; }  // garbage ptr if slot lost
  }
  return agg[0].clicks;
}

PR #1965 fixes all three walkers (collect_vars_recursive, body_contains_yield, scan_stmt_for_max_func) plus linearize_body (do-while → flagged-while desugar; labeled loop → linearize the wrapped loop with its break/continue label rewritten). New regression test byte-compares labeled / labeled-break-continue / do-while / object-accumulator cases against node.

Two things I'd ask, to confirm whether this is the crash you're hitting:

Does the real buildSummaryCache (or anything on the ~260-op path) use a labeled loop (outer: for … continue outer) or a do { … } while around an await? That's the exact trigger. DB-aggregation scans with continue outer are a common shape, and your simplified repro had plain for loops (which do work — they don't hit this), which fits the "minimal repro completes cleanly" observation.
If you can rebuild from PR fix(async): lower awaits inside labeled and do-while loops (#1824) #1965 (or main once it lands) and re-run the gscmaster-api job, does the SIGSEGV survive? You offered to test patches against the deterministic service — that would be the decisive check.

If it still crashes with no labeled/do-while loop involved, then there's a separate cumulative bug and I'll keep digging (the double js_promise_run_microtasks in your backtrace — a nested microtask drain inside the Fastify pump — is my next suspect). But this one is real and in the right neighborhood regardless.

fix(async): lower awaits inside labeled and do-while loops (#1824) (#…

Reopening: `f9d485e` (#1965) does not fix this case — still crashes, identical backtrace

Rebuilt main @ f9d485e0 ("fix(async): lower awaits inside labeled and do-while loops (#1824)") and redeployed the service. The crash persists, with a byte-for-byte identical faulting backtrace (fresh gdb capture on the new build):

#0 js_object_get_field_by_name        (rdi=0x200000000  ← garbage object ptr; rsi = valid field-name str)
#1 js_object_get_field_by_name_f64
#2 perry_closure_<module>             ← resumed continuation
#3 async_step_fulfill_thunk
#4 js_promise_run_microtasks
#5 buildSummaryCache                  ← the async fn
#10 js_stdlib_process_pending / #11 js_run_stdlib_pump / #12 main

The fix targeted labeled and do-while loops, but the offending loop here is a plain for loop with an await followed by a continue, maintaining a long-lived heap accumulator across iterations:

const dDates: string[] = [];
const dAgg: any[] = [];                              // long-lived, captured across every await
for (let i = 0; i < sites.length; i++) {
  const dailyStr = await getCacheEntry(...);         // FFI-async await (a @perryts/mysql SELECT)
  if (dailyStr === null) continue;                   // continue AFTER the await
  const dailyArr = JSON.parse(dailyStr);
  for (let j = 0; j < dailyArr.length; j++) {
    const row = dailyArr[j];
    // linear-scan dDates for idx …
    const existing = dAgg[idx];
    existing.clicks = existing.clicks + (row.clicks || 0.0);   // ← faults: existing/row is the garbage ptr
    // else: dAgg.push({...})
  }
}

Repro status

A standalone repro of exactly this shape — plain for, await fetch() then continue, the same linear-scan + parallel-array accumulator, 8 passes × 37 iterations — completes cleanly on f9d485e0. So the bare pattern isn't sufficient. In the real service it's deterministic; the differences are:

the await is a @perryts/mysql query (getCacheEntry), not fetch;
it runs after a large cumulative workload in the same async invocation (~260 awaited fetch + ~230 mysql ops before this function), and crashes early in the 2nd of three passes.

So the trigger seems to need the mysql-await path and/or the accumulated heap/async state, not just the loop shape.

Offer

I have the symbolized binary and can attach a core dump, disassemble js_object_get_field_by_name + the resumed closure's call site, or wire up a @perryts/mysql-backed minimal repro / test a patch against the deterministic service. Given you just touched this exact codegen, what would help most? Reopening since the labeled/do-while fix doesn't cover this for+await+continue+accumulator variant.

Update: symbolized core dump on f9d485e0 — with the #1824 labeled/do-while fix AND #1957 both present, the crash persists. Confirmed as a mis-restored / uninitialized async-frame spill slot.

Setup: a Fastify service compiled with Perry at f9d485e0. The crash is deterministic ~50s into a background job that does ~260 sequential fetchWithAuth + @perryts/mysql pool.execute calls across 3 date ranges, ending in buildSummaryCache — an async fn with await inside a for loop plus a long-lived parallel-array accumulator.

Symbolized backtrace (gdb on a PERRY_DEBUG_SYMBOLS=1 build):

#0 js_object_get_field_by_name        cmpb $0x5,-0x8(%rdi)   rdi=0x200000000
#1 js_object_get_field_by_name_f64
#2 perry_closure_lib_background_refresh_ts__29        <- resumed await-continuation
#3 perry_runtime::promise::async_step::async_step_fulfill_thunk
#4 js_promise_run_microtasks
#5 perry_fn_lib_background_refresh_ts__buildSummaryCache
#6 perry_closure_lib_background_refresh_ts
#7 js_promise_run_microtasks
#8 perry_stdlib::fastify::server::wait_for_promise
#9 perry_stdlib::fastify::server::js_fastify_process_pending
#10 js_stdlib_process_pending
#11 js_run_stdlib_pump
#12 main

What faults: the resumed continuation closure __29 (the body after const rows = await getCacheEntry(...), i.e. return rows[0].data) reads field "data" from an object pointer that is 0x200000000 — the small denormal an uninitialized slot decodes to (cf. the closure/registry.rs note: "uninit headers slot evaluated to a small denormal float"). It is not a heap address; live heap objects in this dump are at 0x3e49b….

Register / stack state at the fault:

r14 = r15 = 0x200000000; the crashing call is mov %r15,%rdi; mov %rbx,%rsi; call js_object_get_field_by_name (rbx = the interned "data" string).
0x200000000 also occupies several stack spill slots (e.g. …780e0 holds two consecutively).
__29 is a correct 2-arg closure (args in rdi/xmm0/xmm1; invoked via js_closure_call2 from async_step_fulfill_thunk) with no frame pointer — rbp holds a heap ClosureHeader ("CLOS" magic). So this is not an arity-dispatch mismatch; the garbage is loaded from a spilled live-value slot of the resumed frame.

Ruled out:

fix(mysql2): build query result on the main thread, not the tokio blocking worker (worker-thread JSValue construction is UB) #1957 (mysql2 builds the result on the main thread via resolve_with) — merged as 5b5c2fc3, present in this binary; the result set is main-allocated, not a worker-arena dangle.
SIGSEGV: async frame slot mis-restored after await in a loop (js_object_get_field_by_name on garbage ptr; not GC) #1824 labeled/do-while lowering (f9d485e0) — present; our loop is a plain for with await + continue.
GC — PERRY_GC_DIAG=1 reports no GC before the crash; survives PERRY_GEN_GC=0, PERRY_GEN_GC_EVACUATE=0, PERRY_GC_FORCE_EVACUATE=1.

Reproduction: I built 7 standalone reproducers faithfully replicating the shape — await in a for loop with continue, a long-lived parallel-array accumulator, ~260 fetch + ~260 pool.execute, the Fastify microtask pump, large-JSON .json() churn, and ~14 live values across the await (high register pressure). All pass on f9d485e0. The fault only manifests inside the full service after the cumulative ~260-fetch + mysql-write + 3-range workload, so it appears to depend on accumulated runtime/spill state rather than the local function shape alone.

I have a 1.4 GB core + the symbol binary and can run any gdb commands you want, share the core, or provide the exact buildSummaryCache source + Perry's emitted output for __29.

Standalone reproducer

I managed to reduce the gscmaster-api crash to a self-contained reproducer that SIGSEGVs 3/3 on current main (f9d485e0 — includes #1957 and the labeled/do-while loop fix).

The trigger needs three things together (each of which, alone, does not crash):

A real fetch + await response.json() of a large payload — a true async suspension through the pump plus a large native result build. (Reproducers that fake the await with JSON.parse/new Promise(r=>r()) never crash — they resolve synchronously and don't suspend.)
The buildSummaryCache shape: await getCacheEntry(...) (a mysql SELECT via the deferred-resolution path) in a loop, feeding a long-lived parallel-array accumulator mutated across the awaits. fetch+json+getCacheEntry without this aggregation does not crash.
A retained large allocation (const BIG = makeBig(8000)) standing in for the service's resident heap. It shifts the GC trigger cadence so a collection reliably fires during the vulnerable async-resume window; without it the crash is timing-flaky, with it 3/3.

Run

DB_HOST=127.0.0.1 DB_PORT=3306 DB_NAME=perrytest DB_USER=root DB_PASS=secret \
  perry compile perry_1824_repro.ts -o repro && ./repro
# in another shell:
curl http://127.0.0.1:3096/run     # the server process SIGSEGVs partway through

Needs any blank mysql (it creates + seeds its own data_cache table) and outbound internet (fetches a large public JSON; any large-JSON URL works).

Reproducer

// Repro for PerryTS/perry#1824 — SIGSEGV (garbage pointer) in the async
// runtime under heavy `fetch`+`response.json()` interleaved with mysql2
// deferred-result resolution, after a few hundred cumulative async ops.
//
// Run:
//   DB_HOST=127.0.0.1 DB_USER=root DB_PASS=secret DB_NAME=perrytest \
//     perry compile perry_1824_repro.ts -o repro && ./repro
//   curl http://127.0.0.1:3096/run     # process SIGSEGVs partway through
//
// Needs: a reachable mysql (any blank DB — the repro creates + seeds its own
// table) and outbound internet (fetches a large public JSON to force a real
// async suspension + a large native response.json() build, like a GSC reply).
import Fastify from "fastify";
import mysql from "mysql2/promise";

function env(name: string, fallback: string): string {
  const v = process.env[name];
  return (v !== null && v !== undefined && v.length > 0) ? v : fallback;
}
const pool = mysql.createPool({
  host: env("DB_HOST", "127.0.0.1"),
  port: parseInt(env("DB_PORT", "3306"), 10),
  database: env("DB_NAME", "perrytest"),
  user: env("DB_USER", "root"),
  password: env("DB_PASS", ""),
  waitForConnections: true, connectionLimit: 10, maxIdle: 10, idleTimeout: 60000, queueLimit: 0,
});

async function query<T>(sql: string, params?: (string | number | null)[]): Promise<T[]> {
  const [rows] = await pool.execute(sql, params);
  return rows as T[];
}
async function execute(sql: string, params?: (string | number | null)[]): Promise<{ affectedRows: number; insertId: number }> {
  const [result] = await pool.execute(sql, params);
  const { affectedRows, insertId } = result as mysql.ResultSetHeader;
  return { affectedRows, insertId };
}
function uuid(): string { return "r" + Math.floor(Math.random() * 1000000000) + "x" + Math.floor(Math.random() * 1000000000); }

async function getCacheEntry(userId: string, cacheKey: string): Promise<string | null> {
  const escapedKey = cacheKey.replace(/'/g, "''");
  const escapedUserId = userId.replace(/'/g, "''");
  const sql = "SELECT data FROM data_cache WHERE user_id = '" + escapedUserId + "' AND cache_key = '" + escapedKey + "'";
  const rows = await query<{ data: string }>(sql, []);
  if (rows.length === 0) return null;
  return rows[0].data;
}
async function saveCacheEntry(userId: string, cacheKey: string, data: string): Promise<void> {
  const id = uuid();
  const escapedKey = cacheKey.replace(/'/g, "''");
  const escapedData = data.replace(/'/g, "''");
  const escapedUserId = userId.replace(/'/g, "''");
  const sql = "INSERT INTO data_cache (id, user_id, cache_key, data, fetched_at)"
    + " VALUES ('" + id + "', '" + escapedUserId + "', '" + escapedKey + "', '" + escapedData + "', NOW())"
    + " ON DUPLICATE KEY UPDATE data = '" + escapedData + "', fetched_at = NOW()";
  await execute(sql, []);
}
async function buildSummaryCache(userId: string, limitedSites: any[], userPlan: any, dateRange: string): Promise<void> {
  let totalClicks = 0.0; let totalImpressions = 0.0; let totalCtrSum = 0.0; let totalPosSum = 0.0; let totalTrendSum = 0.0;
  for (let i = 0; i < limitedSites.length; i++) {
    totalClicks += limitedSites[i].clicks; totalImpressions += limitedSites[i].impressions;
    totalCtrSum += limitedSites[i].ctr; totalPosSum += limitedSites[i].position; totalTrendSum += limitedSites[i].trend;
  }
  const sc = limitedSites.length;
  const dDates: string[] = []; const dAgg: any[] = [];
  for (let i = 0; i < limitedSites.length; i++) {
    const dailyKey = "daily:" + dateRange + ":" + limitedSites[i].siteUrl;
    const dailyStr = await getCacheEntry(userId, dailyKey);
    if (dailyStr === null) continue;
    const dailyArr = JSON.parse(dailyStr);
    for (let j = 0; j < dailyArr.length; j++) {
      const row = dailyArr[j];
      const date = row.date + "";
      let idx = -1;
      for (let k = 0; k < dDates.length; k++) { if (dDates[k] === date) { idx = k; break; } }
      if (idx >= 0) {
        const existing = dAgg[idx];
        existing.clicks = existing.clicks + (row.clicks || 0.0);
        existing.impressions = existing.impressions + (row.impressions || 0.0);
        existing.ctrSum = existing.ctrSum + (row.ctr || 0.0);
        existing.posSum = existing.posSum + (row.position || 0.0);
        existing.count = existing.count + 1.0;
      } else {
        const f: any = {};
        f.clicks = row.clicks || 0.0; f.impressions = row.impressions || 0.0;
        f.ctrSum = row.ctr || 0.0; f.posSum = row.position || 0.0; f.count = 1.0;
        dDates.push(date); dAgg.push(f);
      }
    }
  }
  const daily: any[] = [];
  for (let i = 0; i < dAgg.length; i++) {
    const dVal = dAgg[i]; const d: any = {};
    d.date = dDates[i]; d.clicks = dVal.clicks; d.impressions = dVal.impressions;
    d.ctr = dVal.count > 0.0 ? dVal.ctrSum / Number(dVal.count) : 0.0;
    d.position = dVal.count > 0.0 ? dVal.posSum / Number(dVal.count) : 0.0;
    daily.push(d);
  }
  daily.sort((a: any, b: any) => a.date.localeCompare(b.date));
  const resp: any = {}; const totals: any = {};
  totals.clicks = totalClicks; totals.impressions = totalImpressions;
  totals.ctr = sc > 0 ? totalCtrSum / Number(sc) : 0.0;
  totals.position = sc > 0 ? totalPosSum / Number(sc) : 0.0;
  totals.trend = sc > 0 ? Math.round((totalTrendSum / Number(sc)) * 10.0) / 10.0 : 0.0;
  resp.totals = totals; resp.daily = daily; resp.siteCount = sc;
  resp.plan = userPlan.plan; resp.siteLimit = userPlan.siteLimit;
  await saveCacheEntry(userId, "summary:" + dateRange, JSON.stringify(resp));
}

// Retained heap pressure: a large, long-lived allocation that stands in for
// the real service's resident heap (auth tokens, site lists, cached payloads).
// It shifts the GC trigger cadence so a collection reliably fires during the
// vulnerable async-resume window below — without it the crash is timing-flaky.
function makeBig(n: number): string {
  const arr: any[] = [];
  for (let i = 0; i < n; i++) {
    const o: any = {};
    o.keys = ["search query number " + i + " with some padding to add bytes"];
    o.clicks = i * 1.0; o.impressions = i * 37.0; o.ctr = 0.0213; o.position = 12.5;
    arr.push(o);
  }
  return JSON.stringify(arr);
}
const BIG = makeBig(8000);

// A real network fetch (forces async suspension) + a large native
// response.json() build — stands in for a GSC searchAnalytics reply.
async function fetchStatsLike(): Promise<number> {
  const r = await fetch("https://jsonplaceholder.typicode.com/photos", { method: "GET" });
  const arr = await r.json();
  let s = 0.0;
  for (let k = 0; k < arr.length; k++) { s = s + arr[k].id; }
  return s;
}
async function processRangeLike(userId: string, sites: any[], plan: any, range: string): Promise<void> {
  const ndays = range === "7d" ? 7 : (range === "28d" ? 28 : 90);
  for (let i = 0; i < sites.length; i++) {
    await fetchStatsLike();
    const arr: any[] = [];
    for (let d = 0; d < ndays; d++) {
      const e: any = {};
      e.date = "day-" + (d < 10 ? "0" + d : "" + d);
      e.clicks = d * 1.0; e.impressions = d * 10.0; e.ctr = 0.02; e.position = 11.1;
      arr.push(e);
    }
    await saveCacheEntry(userId, "daily:" + range + ":" + sites[i].siteUrl, JSON.stringify(arr));
  }
  await buildSummaryCache(userId, sites, plan, range);
}
async function ensureTable(): Promise<void> {
  await execute("CREATE TABLE IF NOT EXISTS data_cache ("
    + " id VARCHAR(64) PRIMARY KEY,"
    + " user_id VARCHAR(64) NOT NULL,"
    + " cache_key VARCHAR(255) NOT NULL,"
    + " data LONGTEXT,"
    + " fetched_at DATETIME,"
    + " UNIQUE KEY uq_user_key (user_id, cache_key))", []);
}

const app = Fastify({ logger: false });
app.get("/run", async (request, reply) => {
  await ensureTable();
  if (BIG.length < 0) { return { ok: false }; } // keep BIG live
  const userId = "perry-1824-repro";
  const sites: any[] = [];
  for (let i = 0; i < 37; i++) {
    const s: any = {};
    s.siteUrl = "sc-domain:site" + i + ".example.com";
    s.clicks = i * 10.0; s.impressions = i * 100.0; s.ctr = 0.02; s.position = 12.3; s.trend = 1.5;
    sites.push(s);
  }
  const plan: any = {}; plan.plan = "pro"; plan.siteLimit = 20;
  const ranges = ["7d", "28d", "90d"];
  for (let r = 0; r < ranges.length; r++) {
    await processRangeLike(userId, sites, plan, ranges[r]);
  }
  return { ok: true };
});
app.listen({ port: parseInt(env("PORT", "3096"), 10), host: "127.0.0.1" }, (err: any, address: any) => {
  console.log("listening " + address);
});

Crash signature

Site and value vary run-to-run (op-count / GC-timing driven), always a garbage pointer in the promise/microtask machinery:

# microtask runner calls a continuation whose code pointer is garbage:
#0  0x0000002600000026 in ??              <- rip jumped to garbage
#1  js_promise_run_microtasks
#2  perry_stdlib::fastify::server::wait_for_promise
#3  perry_stdlib::fastify::server::js_fastify_process_pending
#4  js_stdlib_process_pending
#5  js_run_stdlib_pump
#6  main

# or, the original gscmaster variant — a resumed `getCacheEntry` continuation
# reads its frame-captured `rows` as garbage and faults on `rows[0].data`:
#0  js_object_get_field_by_name           rdi=0x200000000  (field "data")
#1  js_object_get_field_by_name_f64
#2  perry_closure_lib_..._ts              (getCacheEntry continuation)
#3  async_step::async_step_fulfill_thunk
#4  js_promise_run_microtasks
#5  ..._buildSummaryCache
...

The garbage values seen across runs (0x200000000, 0x2800000203, 0x2600000026, …) are all small "denormal-shaped" bit patterns — i.e. uninitialized/corrupted async-runtime pointers (cf. the closure/registry.rs note that an uninit slot evaluates to a small denormal). Not fixed by #1957 (result built on the main thread), the labeled/do-while loop fix, or GC flags (PERRY_GEN_GC=0, PERRY_GEN_GC_EVACUATE=0, PERRY_GC_FORCE_EVACUATE=1). I also have a 1.4 GB core + symbol binary from the production crash if useful.

fix(gc): scope inline-arena resync to the Eden arena (#1824) (#2134)

Update: #2134 verified on the standalone repro, gscmaster still crashes byte-identically — diagnostic narrowing

Pulled current main (58219715 — #2134 in place), rebuilt cargo build --release (full staticlibs, no auto-optimize cache):

Standalone repro from the previous comment (perry_1824_repro.ts, the one that was 3/3 reliable pre-fix(gc): scope inline-arena resync to the Eden arena (#1824) #2134): 0/3 crashes under PERRY_GEN_GC=0 ✓. The fix fix(gc): scope inline-arena resync to the Eden arena (#1824) #2134 lands as advertised.
Production gscmaster service built with --no-auto-optimize (forcing it to link the freshly-rebuilt libperry_stdlib.a with fix(gc): scope inline-arena resync to the Eden arena (#1824) #2134), PERRY_GEN_GC=0 in env (verified via /proc/$PID/environ): 2/2 CRASHES byte-identically to before fix(gc): scope inline-arena resync to the Eden arena (#1824) #2134.

Full symbolized backtrace (same as pre-#2134 modulo address randomization):

#0  js_object_get_field_by_name        rdi=0x200000000   field "data"
#1  js_object_get_field_by_name_f64
#2  perry_closure_lib_background_refresh_ts__29     ← getCacheEntry's resumed continuation
#3  perry_runtime::promise::async_step::async_step_fulfill_thunk
#4  js_promise_run_microtasks
#5  perry_fn_lib_background_refresh_ts__buildSummaryCache
#6  perry_closure_lib_background_refresh_ts__65
#7  js_promise_run_microtasks
#8  perry_stdlib::fastify::server::wait_for_promise
#9  perry_stdlib::fastify::server::js_fastify_process_pending
#10 js_stdlib_process_pending
#11 js_run_stdlib_pump
#12 main

Heavier standalone repros tried — none reproduce post-#2134

Wrote four heavier variants of the original repro, all run 3× under PERRY_GEN_GC=0 on 58219715:

variant	what it adds vs the light repro	result
heavy (`repro_heavy.ts`)	4 fetches/site (mimics fetchSiteStats's multi-call pattern)	3/3 ALIVE
auth-heavy (`repro_auth.ts`)	the above + `fetchWithAuth` (perry builtin) instead of plain `fetch`	3/3 ALIVE
verbatim-buildSummaryCache	verbatim copy of gscmaster's `buildSummaryCache` + write-then-read per range	3/3 ALIVE
token-refresh-mimic	the above + per-site `SELECT/POST/UPDATE` cycle mimicking OAuth token refresh	3/3 ALIVE

So the gscmaster trigger needs something my synthetic shapes can't isolate.

What I instrumented — where `0x200000000` does NOT come from

Added eprintln! + std::backtrace::Backtrace::force_capture() at each of these sites, conditional on the value bits being 0x200000000. Ran each against gscmaster's deterministic crash path (rebuilt + redeployed with --no-auto-optimize each time, ~10 min cycles). For each site below, zero fires out of the 3 crashes — confirming the value does not flow through that path:

runtime_store_jsvalue_slot (the central GC-traced slot store, used by js_object_set_field, store_array_slot, etc.) — 0 fires
js_array_push_f64 (the raw ptr::write element path) — 0 fires
js_promise_resolve at function entry — 0 fires
js_promise_resolved's fast-primitive path (the only direct (*promise).value = ... write in the codebase) — 0 fires
js_async_step_chain's let unwrapped = unsafe { (*inner).value }; (the bypass-via-already-fulfilled-inner path) — 0 fires
async_step_fulfill_thunk at entry (the immediate caller of the crashing closure) — 0 fires
js_object_get_field_by_name at entry (the crash site, where I see it) — 3 fires per refreshUserData run, always with the backtrace above

So 0x200000000:

isn't being written to any GC-traced slot,
isn't being written to a promise.value field,
isn't being passed as a Task::AsyncStep / Promise / Inline value through the microtask runner,
isn't being passed as async_step_fulfill_thunk's value arg,
but is what rows[0] reads in the resumed getCacheEntry continuation.

That narrows it hard: either it's a codegen-inline array index-get (skipping the runtime call layer entirely — the IndexGet codegen-side fast path), or a write through a path my hooks didn't cover (a direct ptr::write in array growth/relocation, write-barrier path, etc.).

Practical workaround verified

The decisive observation: I added a guard at the entry of js_object_get_field_by_name:

if (obj as u64) == 0x200000000 {
    return JSValue::undefined();
}

With this guard in place (no other changes), gscmaster's refreshUserData completes all 3 date ranges cleanly: trigger returns 200, 0 restarts, "Completed refresh for user …" logged. The guard fires exactly 3 times per run (once per range, all with the backtrace above). So masking the bad pointer at the read site is sufficient for the service to function — the bug really is just this specific 0x200000000 leaking into one specific JS object-pointer flow.

What I didn't try (would have been the next move)

The path-of-elimination points at codegen-inline IndexGet reads (which would bypass js_array_get_f64 and js_array_get_jsvalue). I didn't instrument the codegen-emitted inline loads because that's at LLVM-IR emission, not a runtime function I could patch. If you have a knob to disable the inline IndexGet fast-path / force the runtime-call form, that'd be the cleanest discriminator.

Happy to push the instrumented branch, share the diff against 58219715, or run any specific hook you want against gscmaster's deterministic crash path. The standalone repro perry_1824_repro.ts from the previous comment remains the best minimal example for the bug class #2134 fixed.

fix(sharp): toBuffer() resolves a real Buffer instead of a base64 str…

feat(sharp): metadata/toFile objects, Buffer-input factory, .extract(…

Merge lane turnloop/b-db-decline: the agent-post C ABI, and ioredis o…

Navigation Menu

Repository navigation

SIGSEGV: async frame slot mis-restored after `await` in a loop (js_object_get_field_by_name on garbage ptr; not GC) #1824

Summary

Shape of the workload

Environment

Asks

Update: symbolized backtrace — async frame slot mis-restored after an `await` in a loop

Faulting backtrace

What the code is doing

Confirmed NOT the cause (all tried, crash unchanged)

Reproduction notes

Offer

Found and fixed a concrete frame-loss bug in this symptom class — PR #1965

Reopening: `f9d485e` (#1965) does not fix this case — still crashes, identical backtrace

Repro status

Offer

Standalone reproducer

Run

Reproducer

Crash signature

Update: #2134 verified on the standalone repro, gscmaster still crashes byte-identically — diagnostic narrowing

Heavier standalone repros tried — none reproduce post-#2134

What I instrumented — where `0x200000000` does NOT come from

Practical workaround verified

What I didn't try (would have been the next move)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SIGSEGV: async frame slot mis-restored after await in a loop (js_object_get_field_by_name on garbage ptr; not GC) #1824

Description

Summary

Shape of the workload

Environment

Asks

Activity

proggeramlug commented on May 26, 2026

proggeramlug commented on May 27, 2026

Update: symbolized backtrace — async frame slot mis-restored after an await in a loop

Faulting backtrace

What the code is doing

Confirmed NOT the cause (all tried, crash unchanged)

Reproduction notes

Offer

proggeramlug commented on May 27, 2026

Found and fixed a concrete frame-loss bug in this symptom class — PR #1965

proggeramlug commented on May 27, 2026

Reopening: f9d485e (#1965) does not fix this case — still crashes, identical backtrace

Repro status

Offer

proggeramlug commented on May 27, 2026

proggeramlug commented on May 27, 2026

Standalone reproducer

Run

Reproducer

Crash signature

proggeramlug commented on May 28, 2026

Update: #2134 verified on the standalone repro, gscmaster still crashes byte-identically — diagnostic narrowing

Heavier standalone repros tried — none reproduce post-#2134

What I instrumented — where 0x200000000 does NOT come from

Practical workaround verified

What I didn't try (would have been the next move)

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

SIGSEGV: async frame slot mis-restored after `await` in a loop (js_object_get_field_by_name on garbage ptr; not GC) #1824

Update: symbolized backtrace — async frame slot mis-restored after an `await` in a loop

Reopening: `f9d485e` (#1965) does not fix this case — still crashes, identical backtrace

What I instrumented — where `0x200000000` does NOT come from