Repository navigation
Faster unpack compression backend - #62
Merged
Merged
Conversation
Abdo-ka
added a commit
to Abdo-ka/MkPFS
that referenced
this pull request
Jun 21, 2026
Adds a --no-progress flag to the unpack command so the extraction progress bar can be silenced for non-interactive use (scripts, CI, piping output to a file). Complements the progress reporting added in PR PSBrew#62 by making the reporter's enabled state respect the flag. Includes a regression test for the disabled path and documents the flag in the README unpack section.
RenanGBarreto
approved these changes
Jun 21, 2026
RenanGBarreto
left a comment
RenanGBarreto
left a comment
Contributor
There was a problem hiding this comment.
Thank you.
LGTM, let me test. and then I will merge
RenanGBarreto
approved these changes
Jun 23, 2026
RenanGBarreto
changed the title
RenanGBarreto
added a commit
that referenced
this pull request
Jun 27, 2026
…folder, deep read (#65) ## Why Producing an exFAT-wrapped game image today depends on the host OS and elevated privileges: it shells out to `newfs_exfat`/`mkfs.exfat`, mounts a loopback/disk image (`losetup`/`hdiutil`/OSFMount), copies files in as root/admin, and unmounts. That means three divergent code paths, external tools that must be installed, and admin rights — and it cannot run unprivileged or on a machine without the right filesystem tooling. It also forces a two-step workflow (build a full `.exfat`, then compress it), so peak disk is roughly double the game size while both the intermediate image and the output exist. This PR removes all of that. exFAT is built and read entirely in pure Python — no mount, no root, no external binaries — identically on macOS, Linux, and Windows, and the folder→`.ffpfsc` path runs in a single streaming pass with no temporary `.exfat` on disk. `mkpfs pack folder ./game ./game.ffpfsc` now wraps the folder in an exFAT and compresses it straight into the `.ffpfsc` in one pass. ## New commands and flags - `pack folder` (default): folder → exFAT → `.ffpfsc` in one pass, no temp image. - `pack folder --raw`: pack the folder directly as PFS (the previous default). - `pack exfat <dir> [out]`: build a raw `.exfat` image (auto-named `<titleId>.exfat`). - `unpack --deep`: extract the files *inside* a wrapped exFAT, not the inner `.exfat`. - `tree --deep`: list the files inside a wrapped exFAT. ## How it works - **Forward-only exFAT serializer**: computes the full layout up front (sizes are known after scanning) and emits the image in strict offset order, so the byte stream feeds straight into the compressor — no intermediate file, flat memory. - **Read-only exFAT parser** over any seekable source, plus a PFSC-backed view so a `.ffpfsc` can be peeled to its inner contents without writing a temp `.exfat`. - **Multicore compression** on the fused path (bounded look-ahead thread pool; zlib-ng releases the GIL), deterministic output. - Inner image is named `<titleId>.exfat` from `sce_sys/param.json`. - OS-generated metadata (`.DS_Store`, `._*`, `__MACOSX`, `Thumbs.db`, …) is excluded consistently across packing, the AMPR index, and verify. ## Disk and speed - Peak disk drops to `source + output` (no inner `.exfat`, no doubling). - Fused compression scales across cores (≈6x vs single-threaded in local tests), byte-identical regardless of worker count. ## Validation - The generated exFAT is accepted by a real PS5 (boot-tested via ShadowMountPlus), is `fsck_exfat`-clean, and mounts/reads correctly under the macOS kernel driver. - Round-trip tested through an independent in-tree exFAT reader; committed tests use a tiny gzipped fixture (no mount, portable). Full suite green. ## Notes - The fused path is unsigned only; use `--raw` for signed PFS images. ## Dependencies This branch is stacked on two PRs and should be merged after both: - #62 - #63
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two performance changes on the IO/compression paths.
Unpack previously decoded every payload twice (the upfront integrity pass, then again while writing) and reported progress once per file. Now
extract_pfs_imagecallsinspect_pfs_image(verify_payloads=False), decoding each payload exactly once while writing, with byte-based throttled progress.Compression backend switches from stdlib
zlibtozlib-ng(drop-in: samecompress/decompress/crc32, RFC-1950 compatible).zlib-ngas a runtime dependency.Output is otherwise unchanged (PFS/PFSC format identical). Verified with round-trip pack→unpack, a stdlib-zlib-decode-compat test, and the existing suite.