Skip to content

Bound PyTorch ZIP storage entry lookups - #41

Merged
leehack merged 3 commits into
mainfrom
fix/torch-zip-entry-lookup-5ef5f1b
Nov 3, 2026
Merged

leehack merged 3 commits into
mainfrom
fix/torch-zip-entry-lookup-5ef5f1b

Conversation

@leehack

@leehack leehack commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

PyTorch ZIP metadata restarts an archive scan for every tensor storage. Each entry open also converts its timestamp, which can exceed 30 seconds on a small synthetic checkpoint before tensor data is read. Cache the first normalized name, index and size as one reader scans forward, bounding lookup scans to one archive traversal while preserving duplicate-name order and tensor byte checks. Related to #24. Accepted macOS PR-artifact diagnostics and pending final-release qualification are recorded separately below.

The maintainer explicitly approved a patch-specific exception for this ZIP lookup patch: the prepared upstream report stays local, and owner #24 tracks retirement until an upstream report is authorized. The general carried-patch policy is unchanged. No upstream issue or report was posted.

At f9a734fe225247566ce4de5445e055172590d241, only that patch-table row differs from 6f54ed2d257faa0ed27d2a93cc5d89ea268e622d; every other tracked object, including the native patch and submodule pin, is byte-identical. Fresh model-free checks pass: eight ZIP lookup regressions (linear operation counts, unpatched controls, semantic/cleanup mutants and ASan/UBSan), five pickle-span tests, and 19 tooling tests with two expected platform skips. No model or provider execution accompanied this documentation update. Renewed independent blocking review accepted this exact head with zero P1/P2 findings (receipt SHA256 4af17a9a204ca513712c7464815117b541d628409a2198406e0a352f75220334). All 17 fresh-head CI checks passed, including the separately rerun iOS simulator job that initially could not acquire a hosted runner; that failure is retained.

Retained metadata evidence from the unchanged runtime fix: original flat 0.235136 s and nested 0.220261 s, each with 1,329 ZIP entries, 1,323 tensors, 2,656 entry opens, zero tensor extractions and verified cleanup. Full name/dtype/shape/byte-count inventories match the freshly hash/header-verified original Safetensors file; the two native inventories match each other. The prior unchanged flat reader exceeded 400,000 opens and timed out at 30 seconds. This metadata evidence proves inventory only; the separate original-mode load/generation diagnostic below provides bounded acceptance for the PR artifact.

Accepted actual diagnostic (existing #1703 SD update): macOS arm64 PR artifact 13249907568 / run 41673639935, owner source 6f54ed2d257faa0ed27d2a93cc5d89ea268e622d, passed all six maintained native/public CPU cases using original Safetensors, flat PyTorch ZIP and nested ZIP with external TAESD. All six raw RGB outputs are byte-identical: SHA256 57cdf29472ff1c6eae8c0eb78deb6d08bc7cf629a0139f43173da655ce16a3a8. Loads took 1.20–5.72 seconds, every case met the original 45-second load and 180-second whole-case bounds, and all 16 owned processes exited zero and were reaped without cleanup errors. Independent review verified exact source/plan/runtime bindings and raw hashes.

Diagnostic binding Exact identity
Maintained harness source 2fdb0cb0474dbcc74c9a69ef067cdd18b7994f1c
Native/public PR library SHA256 3c371373a9d91599c44a1301c3a3b8c3dca629c1b0cbfb3c1cf687bde4a3a5c8
Admitted plan SHA256 8ca417b102e9fdd8ada655177036e441dd0147cec68caddccc26426f50ea8327
Independent actual receipt SHA256 c6c879451cac5d74395ed62a05a8760b20d61f067cf50168e53742014542e52f

This accepts that six-case PR-artifact diagnostic. Qualification of a final merged/release artifact, publication and core adoption remain separate pending gates; this is not published-artifact or broader platform/backend qualification. Earlier failed runs remain retained.

The earlier Linux CI failure was the test harness forcing strict C11 while production uses GNU11. The harness now matches production and exposes bounded compiler diagnostics; production patch bytes did not change.

Source review and CI are complete. Final merged/release-artifact qualification remains a separate owner-publication gate; the upstream report remains local.

@leehack
leehack marked this pull request as ready for review November 3, 2026 20:01
@leehack
leehack merged commit 5fde7dd into main Nov 3, 2026
33 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant