Repository navigation
Native @aws-sdk/client-s3 (+ s3-request-presigner) and fastify multipart/form-data parsing #551
Description
Context
Porting sharity-api (Express + Mongoose, ~23k LoC) to Perry. Two
adjacent gaps block every upload flow: server-side multipart parsing
and AWS S3.
Multipart / form-data on fastify
The Fastify server in perry-ext-fastify handles JSON bodies, but every
upload route in the target app is multipart/form-data — avatars, cover
photos, item images. Need a multer-equivalent or fastify-multipart-style
plugin with:
- Disk and memory storage modes
- File size limits (
limits: { fileSize: 5 * 1024 * 1024 }) - MIME filter (
fileFilter: (file) => /^image\//.test(file.mimetype)) - Multiple field names (
avatar,image,cover-photo) - Generated filenames (UUID + original extension)
- Stream the file body without buffering the whole multipart payload in
memory
AWS S3
Sharity-api uses S3 as the canonical image store. Local disk fallback
exists but isn't viable in production. Surface needed:
@aws-sdk/client-s3:PutObjectCommand,GetObjectCommand,
DeleteObjectCommand,HeadObjectCommand@aws-sdk/s3-request-presigner:getSignedUrl(command, {expiresIn})- Region + credential resolution from env (
AWS_REGION,
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_S3_BUCKET_NAME) - ACL support (
public-readis what the existing app uses)
Acceptance
A fastify route that accepts a multipart upload, streams it to S3 with
PutObjectCommand, and returns a presigned GET URL for the uploaded key.
Compiles and runs against real S3 (or LocalStack) under perry run.
Why both in one issue
The two together are how every upload endpoint in a typical Node API is
wired. Splitting them lets the multipart side ship without S3 (still
useful for disk-only setups), but the issue tracks both because
sharity-api needs both.
Activity
proggeramlug commentedon May 7, 2026
Researched the pure-JS S3 ecosystem before sketching a
perry-ext-s3Rust binding. The best fit is@bradenmacdonald/s3-lite-client(MIT, zero deps, ~1.9k LoC across 4 runtime files).Audited it against Perry's TS-subset:
eval/new Function/ dynamic importsSymbol(...)/Proxy/WeakMap/ReflectBuffer/node:stream/node:crypto— Web APIs onlySurface coverage:
PutObject,GetObject,DeleteObject,statObject(Head),copyObject,listObjects,presignedGetObject/presignedPutObject/presignPostV4, multipart streaming uploads, ACL (x-amz-acl: private | public-read | …). Matches everything #551 lists.Two stdlib gaps block native compilation today:
crypto.subtle.digest/importKey/sign(HMAC-SHA-256, SHA-256) — Web Crypto: implement crypto.subtle.digest / importKey / sign (HMAC-SHA-256, SHA-256) #561WritableStream/TransformStream— Web Streams: allow user classes to extend WritableStream / TransformStream / ReadableStream #562Both are general-purpose primitives many other packages will need. Read paths (
getObject,statObject,deleteObject,listObjects,presignedGetObject,presignedPostObject) hit only #561; the streaming upload path needs #562 too.With both landed,
s3-lite-clientshould compile viaperry.compilePackageswith no patches. Bonus: it speaks any S3-compatible endpoint (R2, MinIO, B2, Spaces, Supabase, LocalStack) for free — aperry-ext-s3Rust binding would lock you into AWS-only unless endpoint flexibility is re-implemented.Multipart/form-data parsing on fastify is the orthogonal half of #551 and not addressed here — that's
@fastify/multipart(busboy-based) territory, separate research.proggeramlug commentedon May 7, 2026
First compile attempt at v0.5.655 (post #561 + #562) — got 9 of 10 modules through HIR + codegen. Three new gaps surfaced underneath, all with reproducible repros:
String.prototype.split(sep, limit)2-arg form rejected by codegen. Hitssigning.ts:254which useselement.split("=", 2)for SigV4 query-string canonicalization. Hard blocker.export { X as Y }rename produces undefined-symbol link error #568 —export { Client as S3Client }rename produces undefined-symbol link error. Source-side emitsClient__greet, consumer-side callsS3Client__greet, link fails. Hard blocker, structural (not s3-lite-specific).functiondeclarations not hoisted — silent runtime undefined #569 — Function-scopedfunctiondeclarations not hoisted at HIR lowering.xml-parser.ts(recursive-descent parser shape, ~10 inner functions) compiles with warnings then returns undefined at runtime. Silent runtime corruption — every S3 XML response would parse to garbage.With those three landed, this should compile + run for at least the read paths (Get/Head/Delete/List/presigned*).
proggeramlug commentedon May 8, 2026
v0.5.663 (post #567 + #568 + #569): the package compiles and links cleanly. Read-path runtime exposes three more gaps, all with reproducible repros:
0. Top-level functions are fine; class members are broken. Hits S3Client'sconstructor({ endPoint, useSSL, port, pathPrefix, ...params })— every param arrives as 0, the validation chain trips and throws immediately. Hard blocker, structural.class X extends Error {}(bare) loses constructor args + SIGABRT on property access #573 —class X extends Error {}(bare-derived) drops thesuper(msg)arg and produces an instance with.name/.messageundefined; intermittent SIGABRT on first property access. Hits the entire s3-lite error hierarchy (S3Error → InvalidArgumentError → InvalidEndpointError → ...).import * as Ns from "./mod.js"produces non-object namespace; class members unreachable #574 —import * as Ns from "./mod.js"produces a non-object namespace where class exports are unreachable (primitive consts partially work). Hitsexport * as S3Errors from "./errors.js"and breaks everyinstanceof S3Errors.ServerErrorcheck.All three are general-purpose gaps that will hit any modern TS-style npm package, not just s3-lite. With them landed I'll re-test.
proggeramlug commentedon May 8, 2026
v0.5.664 (post #572 + #573 + #574): the package compiles, links, runs, and produces a presigned URL end-to-end. Constructor works, all imports resolve, the SigV4 chain runs without crashing, and a URL string comes back. Big jump.
The URL is structurally correct but has two correctness bugs:
URLSearchParams(...).toString()returns"[object Object]"for every input shape (empty record, populated record, query string). HitsbuildRequestOptionsand injects the stray?[object Object]=&segment. Beyond s3-lite, breaks every form-encoder / URL builder.for…of headers.keys()produces empty iteration;headers.forEachworks #576 —for (const k of headers.keys())iterates zero entries.headers.forEach((v, k) => ...)correctly yields all of them. The SigV4 canonical-request build usesfor…of headers.keys()→ SignedHeaders is empty → signature is empty → the URL would fail authentication against AWS even though the structure is right.With those two landed, the URL should match
bunbyte-for-byte and authenticate against real S3. Read paths (Get/Head/Delete/List/presign) should then work. Upload path needs the cross-module TsParamProp slot bug separately — that one isn't reached yet by the smoke test.fix(hir+codegen): closes #572 #573 #574 — class destructure params + …
fix(stdlib+hir+codegen): closes #575 #576 — URLSearchParams.toString(…
proggeramlug commentedon May 8, 2026
v0.5.665 (post #575 + #576): the produced presigned URL is now structurally identical to bun byte-for-byte except the signature itself. Same query params, same order, same encoding. The 64-char
X-Amz-Signature=...field is the only mismatch.Compared:
Two new issues, both in the typed-array runtime:
for…of typedArray/Array.from(typedArray)/[...typedArray]all yield NaN-boxed bit-pattern garbage instead of byte values.bin2hexoutput is wrong even though the underlying HMAC bytes are correct (verified via index-based access).new Uint8Array(arrayBuffer)does not produce a live view; second view of same ArrayBuffer reads zeros #579 —new Uint8Array(arrayBuffer)does not alias the underlying buffer; second view reads zeros. Hits Web Crypto'sArrayBuffer-returning APIs, breaks every signed-URL chain.Fix either of these alone won't unblock — both have to land. With both, the resulting URL should authenticate against real S3.
fix(runtime+hir+codegen): closes #578 #579 — typed-array iteration + …
proggeramlug commentedon May 8, 2026
v0.5.668 (post #578 + #579): typed-array iteration + ArrayBuffer aliasing both verified working. The s3-lite presigned URL would now match bun byte-for-byte EXCEPT a separate codegen regression from v0.5.667 (#536) blocks the compile:
js_string_addrefcalled but not declared as extern; clang fails on everylet y = x; ... finallypattern #580 — `js_string_addref` is called from codegen but never declared as an extern. Any `try { const y = x; ... } finally { ... }` with string-typed locals fails clang -c with `use of undefined value '@js_string_addref'`. Fix is a 1-line declaration in `runtime_decls.rs` next to the existing `js_get_string_pointer_unified` entry. The package's `signing.ts` hits this pattern in several places.Once #580 lands the read path should compile cleanly. The byte-for-byte URL match is essentially guaranteed since (a) every step up to the signature is already byte-identical and (b) the typed-array fix verifies the HMAC bytes pass through correctly to bin2hex.
fix(transform+codegen): closes #536 — try { return X } finally { Y } …
proggeramlug commentedon May 8, 2026
v0.5.676 (post #580): the package compiles, links, runs, and produces a presigned URL with a real 64-char HMAC signature. Every URL field except the signature itself is byte-identical to bun.
Both well-formed (32-byte HMAC), so the crypto chain works end-to-end. They differ because the canonical request's path segment is corrupted — instrumented signing.ts to dump it:
Root cause traced to a separate runtime bug, filed as:
new Uint8Array(arrayBuffer)does not produce a live view; second view of same ArrayBuffer reads zeros #579 fixed iteration of Uint8Array storage; this is the orthogonal bug in `TextEncoder`'s WRITE path.s3-lite calls TextEncoder twice in the signing chain (once in awsUriEncode, once via sha256digestHex), so the canonical request, string-to-sign, and final HMAC all derive from corrupted bytes. Once #584 lands the URL should be byte-identical to bun.
fix(runtime+hir+codegen): closes #584 — TextEncoder().encode() now pr…
proggeramlug commentedon May 8, 2026
v0.5.690 (post #584): TextEncoder produces correct bytes now, the canonical request's URI segment is correct (`/test-bucket/hello.txt`), and the signature is a real well-formed HMAC. URL is byte-identical to bun except the signature itself:
Instrumented signing.ts again to dump the canonical request. The query-string segment of the canonical request is empty, even though the URL has 5 X-Amz-* params:
Traced to a separate runtime bug:
str.split(sep).map(fn).sort()returns empty array; storing split result in temp fixes it #587 — `str.split(sep).map(fn).sort()` returns an empty array. Storing the split result in a temp first works. `[...split(...)]` spread also works. `Array.from(split(...))` does NOT work. Combo of three things is required: split result + map + sort. Affects every CSV/query-string/header parser shape, not just s3-lite.With #587 landed the URL should be byte-identical to bun and authenticate against real S3.
proggeramlug commentedon May 8, 2026
v0.5.692 (post #587): The presigned URL is byte-for-byte identical to bun's output.
@bradenmacdonald/s3-lite-clientcompiles natively under `perry.compilePackages` (no patches), the full SigV4 signing chain runs end-to-end (TextEncoder → SHA-256 hashing → HMAC-SHA-256 key derivation → final HMAC), and produces a valid AWS-authenticatable URL.Read paths (`presignedGetObject` / `presignedPostObject` / `getObject` / `statObject` / `deleteObject` / `listObjects`) should all work — they share this exact signing chain. The byte-match against bun is the strongest possible verification short of hitting real S3.
Total path from #551 filed → byte-identical URL: 12 issues filed and closed (#561 #562 #567 #568 #569 #572 #573 #574 #575 #576 #578 #579 #580 #584 #587), spanning Web Crypto, Web Streams subclassing, class destructure params, Error inheritance, namespace imports, URLSearchParams, Headers iteration, typed-array iteration + ArrayBuffer aliasing, TextEncoder, and chained array-method evaluation. Each one a general-purpose gap, not s3-lite-specific.
Multipart upload path is the orthogonal remaining concern (would exercise WritableStream subclassing + the cross-module TsParamProp slot bug) — separate test from the read-path completion captured here.
proggeramlug commentedon May 8, 2026
Documentation added — `@bradenmacdonald/s3-lite-client` is now listed as a known-working package and has its own usage section in the HTTP & Networking docs:
Both docs cross-reference this issue.
docs(packages+stdlib): document @bradenmacdonald/s3-lite-client as kn…