Repository navigation
AMPR emulation automated index and OS-metadata exclusion for folder packing - #63
Merged
RenanGBarreto merged 5 commits intoJun 23, 2026
Merged
Conversation
RenanGBarreto
changed the title
RenanGBarreto
added a commit
that referenced
this pull request
Jun 27, 2026
…folder, deep read (#65) ## Why Producing an exFAT-wrapped game image today depends on the host OS and elevated privileges: it shells out to `newfs_exfat`/`mkfs.exfat`, mounts a loopback/disk image (`losetup`/`hdiutil`/OSFMount), copies files in as root/admin, and unmounts. That means three divergent code paths, external tools that must be installed, and admin rights — and it cannot run unprivileged or on a machine without the right filesystem tooling. It also forces a two-step workflow (build a full `.exfat`, then compress it), so peak disk is roughly double the game size while both the intermediate image and the output exist. This PR removes all of that. exFAT is built and read entirely in pure Python — no mount, no root, no external binaries — identically on macOS, Linux, and Windows, and the folder→`.ffpfsc` path runs in a single streaming pass with no temporary `.exfat` on disk. `mkpfs pack folder ./game ./game.ffpfsc` now wraps the folder in an exFAT and compresses it straight into the `.ffpfsc` in one pass. ## New commands and flags - `pack folder` (default): folder → exFAT → `.ffpfsc` in one pass, no temp image. - `pack folder --raw`: pack the folder directly as PFS (the previous default). - `pack exfat <dir> [out]`: build a raw `.exfat` image (auto-named `<titleId>.exfat`). - `unpack --deep`: extract the files *inside* a wrapped exFAT, not the inner `.exfat`. - `tree --deep`: list the files inside a wrapped exFAT. ## How it works - **Forward-only exFAT serializer**: computes the full layout up front (sizes are known after scanning) and emits the image in strict offset order, so the byte stream feeds straight into the compressor — no intermediate file, flat memory. - **Read-only exFAT parser** over any seekable source, plus a PFSC-backed view so a `.ffpfsc` can be peeled to its inner contents without writing a temp `.exfat`. - **Multicore compression** on the fused path (bounded look-ahead thread pool; zlib-ng releases the GIL), deterministic output. - Inner image is named `<titleId>.exfat` from `sce_sys/param.json`. - OS-generated metadata (`.DS_Store`, `._*`, `__MACOSX`, `Thumbs.db`, …) is excluded consistently across packing, the AMPR index, and verify. ## Disk and speed - Peak disk drops to `source + output` (no inner `.exfat`, no doubling). - Fused compression scales across cores (≈6x vs single-threaded in local tests), byte-identical regardless of worker count. ## Validation - The generated exFAT is accepted by a real PS5 (boot-tested via ShadowMountPlus), is `fsck_exfat`-clean, and mounts/reads correctly under the macOS kernel driver. - Round-trip tested through an independent in-tree exFAT reader; committed tests use a tiny gzipped fixture (no mount, portable). Full suite green. ## Notes - The fused path is unsigned only; use `--raw` for signed PFS images. ## Dependencies This branch is stacked on two PRs and should be merged after both: - #62 - #63
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related source-prep changes for folder packing.
AMPR index. When a source folder contains
fakelib/libSceAmpr.sprx(an emulation build),pack foldernow generates anampr_emu.index(AMPRIDX3: FNV-1a-64 path hash table + per-file size/mtime records) into the folder before packing, so it ships inside the image. It's regenerated on every pack to stay in sync with the current files — the rebuild is metadata-only (directory walk + stat, no payload reads), so it's inexpensive.--no-ampr-indexskips it and preserves any existing index.OS-metadata exclusion. A centralized
is_ignored_namehelper now drops OS-generated junk during scanning and AMPR indexing — macOS (.DS_Store,._*,__MACOSX,Spotlight-V100, ...) and Windows (Thumbs.db,desktop.ini,$RECYCLE.BIN, ...) — including whole junk directories. This keeps such files out of every image and the index.Note — behavior change: files like
.DS_Store/Thumbs.dbthat were packed before are now excluded (always on; they're never wanted in a PS5 image).On-disk PFS/PFSC format is unchanged. Adds unit + integration tests for the index format/gating and the exclusion across both code paths.