Investigate large PyTorch ZIP checkpoint load timeouts #24

Current delivery — 2026-11-04

#41's bounded archive lookup is published in v0.3.0-3; scaling/causal and accepted original-container generation evidence. Final Mac runtime bytes match the accepted six-case native/public CPU diagnostic. Main remains -2, public core remains -1, and core draft#1739 stages -3. Keep open/P2 through final integration/adoption and retained acceptance; this does not establish universal layout/platform/time guarantees or retroactively explain every historical timeout. The approved patch-retirement report stays local. Original evidence follows unchanged.

Problem

Full SD-Turbo checkpoints written by standard PyTorch 2.11.0 torch.save in ZIP format did not finish loading within 45 seconds through either published llamadart 0.15.3 or direct native entry points in the same stable-diffusion-native v0.2.0-4 library. Both flat .pt and nested {'state_dict': ...} .ckpt layouts behaved this way.

The cause is unresolved. This is a bounded timeout observation, not proof of an infinite hang, universal PyTorch incompatibility, or a newly established regression between releases. Keep it separate from the confirmed greater-than-2-GiB legacy pickle scanner defect.

Found during llamadart#1218. Runtime pins and upstream sources remained unchanged.

Exact baseline and inputs

• Owner artifact: v0.2.0-4, bundled upstream 3f8527a46c54ecf4cb4ed6003da8e8982283c73c.
• macOS arm64, CPU/four threads, eager loading; published core llamadart 0.15.3.
• Runtime archive SHA-256: 840df6e27e1cac6d378dd5529f7c3b8d3b176417f950ef523c3535c09c92f528.
• Runtime dylib SHA-256: 79ed1ab7c0451596bcb18070d459826a8b04409875873e91f39ffd84b2ffe1ce.
• Trusted full SD-Turbo Safetensors: 1323 tensors (709 F16, 614 F32), 2,612,796,864 bytes; SHA-256 4505ca3da482f3e01679f6301bf23cde136e0cb3ffeb66b7e809038f619a19b4.
• Standard ZIP torch-save.pt: 2,613,072,319 bytes; SHA-256 05e2a2fef17ef03962c522374c01c373a71d5cbde20380f7800678c3ae99a75d.
• Nested ZIP torch-save.ckpt: 2,613,072,319 bytes; SHA-256 52dfdb8aa825c27bded6f11c5f25f8c4fe57405c0fc633e2f74f375186a36309.

Reproduction

Start from trusted full SD-Turbo Safetensors with the same tensor names/values. Build an OrderedDict of tensors by reading its JSON header and raw tensor storage, then use PyTorch 2.11.0:

torch.save(state, 'torch-save.pt', pickle_protocol=2)
torch.save({'state_dict': state}, 'torch-save.ckpt', pickle_protocol=2)

No torch.load, pickle.load, arbitrary checkpoint code execution, or downloaded pickle was used. Handwritten archive attempts were excluded from this result; the files above are standard torch.save output.

Load each file as a checkpoint with a full TAESD decoder, forcing CPU/four threads. The direct C control initializes the published header's parameters as follows:

sd_ctx_params_t cp;
sd_ctx_params_init(&cp);
cp.model_path = checkpoint_path;
cp.taesd_path = full_taesd_safetensors_path;
cp.backend = "cpu";
cp.n_threads = 4;
cp.diffusion_flash_attn = true;
cp.vae_conv_direct = false;
cp.eager_load = true;
sd_ctx_t* ctx = new_sd_ctx(&cp);

Dart uses ImageGenerationEngine.load with ImageModelRole.checkpoint, an explicit taesd component, and ImageModelParams(device: ComputeDevice.cpu, threads: 4). Each test runs in its own subprocess and is terminated after 45 seconds.

Both full-checkpoint native logs ended after format dispatch into init_from_torch_zip_file:

model_loader.cpp:224 - load <checkpoint> using torch zip format
model_loader.cpp:389 - init from <checkpoint>

The pinned parse_torch_state_dict_pickle successfully parses all 1323 tensors from the ZIP metadata alone. That narrows the investigation but does not establish where loading stalls. The native control uses the same published owner binary, not a separately built newer upstream reference.

Current verification — November 2

Recovered all three original input files byte-for-byte, including both recorded ZIP hashes. Nine serialized direct-native load controls passed with matching artifact headers: original v0.2.0-4 loaded flat ZIP in 22.52 s, nested ZIP in 7.13 s and Safetensors in 4.67 s; the prepared v0.3.0-2 macOS artifact loaded them in 13.67/7.38/4.50 s with original fields, and 7.40/7.32/0.83 s with explicit equivalent modes. Every context was freed; none crossed the original 45-second deadline. These are individual observations with uncontrolled OS file-cache warmth, not performance bounds or a benchmark improvement claim.

The original timeout is not reproduced by these direct-native controls, including the unchanged original library. Published llamadart 0.15.3 public load/dispose controls now also pass with those original inputs and the original v0.2.0-4 runtime: flat ZIP loaded in 22.46 s, nested ZIP in 7.05 s and Safetensors in 1.02 s. Each ran in a separate compiled consumer process with CPU/four threads, external 45-second observation and 180-second diagnostic cap; the actual mapped library and all 21 file-backed Mach-O sections were checked against the original artifact. No case crossed 45 seconds. Cache warmth remains uncontrolled. Root cause, same-weight generation and a causal regression test remain unresolved; these non-reproductions are not fix or closure evidence. Hash-bound logs and matching-header harness receipts are retained under sd24-original-inputs/load-controls-results.json in the release evidence.

Acceptance

• Profile the native ZIP load using these trusted large inputs; separate ZIP/container access, metadata parsing and tensor loading costs.
• Determine whether this is an algorithmic performance issue, layout/size limit or another defect; document expected supported layouts and load-time bounds.
• Add a focused native regression or scaling test for the identified cause, preserving successful small-component behavior and malformed-input rejection.
• Repeat full flat/nested ZIP loading and same-weight generation through direct native and public Dart paths after the fix.
• Keep any source/artifact update in the owner workflow; downstream runtime repinning requires separate qualification and authorization.

Downstream docs/regression tracking: llamadart draft PR #1237, with runtime pins unchanged.