[rbuergi]

What & why

The first main run of the moved ollama lane (https://github.com/Systemorph/MeshWeaver.Deploy/actions/runs/40493629457) was cut at 45 minutes inside the push. ocr, whisper and s1-extract published on the same commit, so the identity, the data-plane login and the tag-list read are proven. Only ollama's upload didn't finish.

Where the time went (job Build and push ollama to ACR, step log):

phase	Deploy run (cut)	core's green run 40232022675
free disk space	4.6 min	3.7 min
build (bake.sh 201 s / 182 s)	4.5 min	4.1 min
exporting layers (zstd)	3.9 min	3.1 min
unpacking into the local store	3.9 min	3.0 min
pushing layers (~24 GB model layer)	> 28.1 min, not done	14.7 min (880 s)

Both runs used the same runner type (ubuntu-latest), the same flags and the same registry. Neither had any build cache: core never exported one, and Deploy's BuildKit state starts cold. The other three lanes had finished pushing by 11:15:54Z, and ollama's push began at 11:23:39Z, so they did not compete. The logs do not show why this upload was at least twice as slow: BuildKit prints no per-blob progress. I did not establish that cause.

The root. Every publication rebuilt the image from nothing and re-uploaded the whole ~24 GB weights layer. The layer is not reproducible (file times), and there was no cache to reuse it from. So the job's duration was bound to one huge upload at whatever rate that run got. Raising the bound would not change that, so this PR removes the repeated upload instead.

The change (ollama-image.yml)

• New cache job. It runs --cache-from + --cache-to against a registry cache: type=registry,ref=<acr>/ollama:buildcache-<line>, mode=max, zstd, image-manifest=true, stored in the image's own repository. It uses --output type=cacheonly, so there is no image load and no unpack.
  • Warm cache: every step is CACHED and nothing is uploaded.
  • Cold cache: the weights go up once, in a job that does nothing else.
• publish now needs: cache. It builds --cache-from that cache and pushes by reference. unpack=false: a publish never runs the image, and unpacking cost 3.9 min of the cut run. There is no force-compression: the cached layers are already zstd, and forcing it would fetch every lazy blob to recompress it.
• Unchanged: write-once, the data-plane login, converge-line. check-workflow and check-line-convergence are green, and so are timeouts (35 jobs), YAML keys and secret preflight. actionlint is clean.

Proof (local, not CI)

Setup: Docker 29.5, buildx 0.35, the containerd image store (the lane's shape), registry:2, and a 300 MB random layer standing in for the weights. Run against a fresh, empty registry:

• cold cache job: rc 0, no image loaded or unpacked;
• warm rerun: the big RUN is CACHED;
• publish on a FRESH builder (cache pruned, no local image) with only --cache-from: pushed in 1 s, and the image's big layer is the cache's blob (sha256:13e24934815e…, 314,587,417 bytes, tar+zstd). The bytes are referenced, not re-uploaded;
• negative control: the same build without --cache-from produces a different blob.

Is dispatching after the fix safe?

No dispatch is needed. The merge commit touches ollama-image.yml, so the push trigger runs the lane on it. A dispatch would be safe anyway:

• The image is write-once, and the failed e4cfec6 never wrote its tag (cut before the manifest), so nothing exists to overwrite.
• converge-line only names published tags in main order.
• The first run on this fix is still a cold cache: the cache job uploads the ~24 GB once, after about 12 min of setup, build and compression. That leaves about 33 min for the upload, against 14.7 min on core's run and more than 28.1 min on the cut run. If the upload is again that slow, the cache job is what gets cut, and nothing is published or half-written. Every run after a successful cache upload (new commit, rerun, dispatch) skips the 24 GB.

Not verified

• The ACR side: the cache export and the lazy push were proven against registry:2, not ACR.
• Whether the retention tasks would prune buildcache-<line> (they are disabled or dry-run today). A pruned cache only costs a cold rebuild.
• The cause of the slow upload rate.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

[Copilot]

🔵 Needs a closer look

Cache coordination and rerun recovery issues remain unresolved, and the new publication path has not been validated against ACR.

2 open issues

• Prevent overlapping runs from overwriting the shared cache reference · New
• Check immutable image tags before rebuilding pruned caches · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Registry-cache export and lazy publication have local proof but remain unverified against Azure Container Registry and need human validation.

No open issues 🎉

2 resolved since last review

• Prevent overlapping runs from overwriting the shared cache reference
• Check immutable image tags before rebuilding pruned caches

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The cache guard needs correction, and registry cache export and lazy publication remain unverified against ACR.

1 open issue

• Ensure cache guard rejects mutable refs and enforces nonzero exit · New

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Registry-cache export and lazy image publication have local proof but remain unverified against the target Azure Container Registry.

No open issues 🎉

1 resolved since last review

• Ensure cache guard rejects mutable refs and enforces nonzero exit

_{Review effort: Balanced}