You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
'buildx imagetools create' broken since v0.31.1 when building multi-platform images on github across multiple runners #3708
We noticed, the 'merge' step failed since 6th of February, which was the day docker/setup-buildx-action started to deploy buildx version v0.31.1.
The logs always error at the docker buildx imagetools create step with
#1 [internal] pushing ghcr.io/pi-hole/***
#1 24.69 pushing sha256:1ac73ae4246c36835675145362d146a4bd2af7bd518de3e2bda44cf62116b55d to ghcr.io/pi-hole/***:nightly
#1 DONE 25.1s
ERROR: ghcr.io/pi-hole/***@sha256:f4a9f4daa0088ae6b74c45149690f9c31420122ad9079c231df81b2e31629010: not found
Error: Process completed with exit code 1.
What changed in v0.31.1 is imagetools create copy behavior from #3623: it now copies only the selected descriptor(s), instead of relying on broader copy side effects. Your workflow runs imagetools create twice and reuses all metadata tags (docker hub + GHCR) for both runs; with this pattern, the second run can reference ghcr.io/...@sha256:... digests that are not present in GHCR yet, which matches the not found error you see.
Also, if you want to simplify this flow, consider switching to our GitHub Builder. This keeps multi-platform image publishing to Docker Hub and GHCR while simplifying the repository-specific GitHub Actions logic. Also reduces CI workflow complexity and maintenance burden for multi-platform builds. Compared to our "Distribute build across multiple runners" docs, this removes the need for per-platform matrix orchestration, digest artifact handoff, and a separate manifest merge job.
If you're interested I pushed these changes on a branch: crazy-max/docker-pi-hole@7195273. There is just this build-arg PIHOLE_DOCKER_TAG=${{ steps.meta.outputs.version }} that would not work because metadata outputs are not exposed here but that probably something we could improve in our reusable workflows using handlebars templates.
Thanks for your comprehensive answer. You really went the extra mile!
I now see where and why we have the issue here. I can implement the workaround but your GitHub Builder looks way more interesting! It would free us from a huge maintenance burden, because handling and orchestrating those distributed builds was not fun. I'm looking forward for the implementation of
Some more summary for this issue in case anyone else hits something similar.
imagetools create has 3 possible ways it can create target image:
make a carbon-copy if single arg and no --platform
make a merged copy if multiple sources
make a filtered copy if --platform filter set.
The case that seems to be hit in here is merged copy where sources are image indexes. In this case the source image index is discarded and replaced with the merged index. Before v0.31.1 (possibly after --platform was added and before v0.31.1) we used to have a bug where we still accidentally copied the source index (and descriptors for unused platforms) as well, even though they remained unreferenced by the target image. This was fixed in v0.31.1 .
If there is a valid use case to access the source index but leave it untagged, we could consider an extra option that would allow push-by-digest when creating the target image.
@yubiuser If this makes sense then I think we can close this issue. If you think push-by-digest could be useful, you can open a new issue to track that.
I think we can close the issue. You have been more than helpful.
Once we switch to the GitHub Builder, we don't have a need for push-by-digest. This was only ever a mean to get things done, nothing we loved to deal with.
Activity
thaJeztah commentedon Mar 7, 2026
cc @crazy-max @tonistiigi
crazy-max commentedon Mar 8, 2026
Thanks for your report.
What changed in v0.31.1 is
imagetools createcopy behavior from #3623: it now copies only the selected descriptor(s), instead of relying on broader copy side effects. Your workflow runsimagetools createtwice and reuses all metadata tags (docker hub + GHCR) for both runs; with this pattern, the second run can referenceghcr.io/...@sha256:...digests that are not present in GHCR yet, which matches thenot founderror you see.So this is reproducible with that workflow shape after v0.31.1. The immediate workaround is to split tags per registry (Docker Hub tags for the Docker Hub create call, GHCR tags for the GHCR call), or otherwise ensure each create call uses source digests from the same registry it is targeting. Something similar to: https://github.com/crazy-max/docker-healthchecks/blob/eef77d758b6c8b06220adc3e4b4f11da1eca6a89/.github/workflows/build.yml#L188-L191
Also, if you want to simplify this flow, consider switching to our GitHub Builder. This keeps multi-platform image publishing to Docker Hub and GHCR while simplifying the repository-specific GitHub Actions logic. Also reduces CI workflow complexity and maintenance burden for multi-platform builds. Compared to our "Distribute build across multiple runners" docs, this removes the need for per-platform matrix orchestration, digest artifact handoff, and a separate manifest merge job.
If you're interested I pushed these changes on a branch: crazy-max/docker-pi-hole@7195273. There is just this build-arg
PIHOLE_DOCKER_TAG=${{ steps.meta.outputs.version }}that would not work because metadata outputs are not exposed here but that probably something we could improve in our reusable workflows using handlebars templates.yubiuser commentedon Mar 9, 2026
Thanks for your comprehensive answer. You really went the extra mile!
I now see where and why we have the issue here. I can implement the workaround but your GitHub Builder looks way more interesting! It would free us from a huge maintenance burden, because handling and orchestrating those distributed builds was not fun. I'm looking forward for the implementation of
and then I'll switch.
tonistiigi commentedon Mar 10, 2026
Some more summary for this issue in case anyone else hits something similar.
imagetools createhas 3 possible ways it can create target image:--platform--platformfilter set.The case that seems to be hit in here is merged copy where sources are image indexes. In this case the source image index is discarded and replaced with the merged index. Before v0.31.1 (possibly after
--platformwas added and before v0.31.1) we used to have a bug where we still accidentally copied the source index (and descriptors for unused platforms) as well, even though they remained unreferenced by the target image. This was fixed in v0.31.1 .If there is a valid use case to access the source index but leave it untagged, we could consider an extra option that would allow
push-by-digestwhen creating the target image.@yubiuser If this makes sense then I think we can close this issue. If you think
push-by-digestcould be useful, you can open a new issue to track that.yubiuser commentedon Mar 12, 2026
I think we can close the issue. You have been more than helpful.
Once we switch to the GitHub Builder, we don't have a need for
push-by-digest. This was only ever a mean to get things done, nothing we loved to deal with.crazy-max commentedon Mar 13, 2026
@yubiuser Opened docker/github-builder#142
ci: pin buildx to v0.31.0 to work around GHCR push-by-digest breakage
ci: push elasticsearch arch images by tag instead of push-by-digest
ci(ket-build-image): build multi-arch (amd64 + arm64) images
ci(ket-build-image): build multi-arch (amd64 + arm64) images
ci(ket-build-image): build multi-arch (amd64 + arm64) images
ci(ket-build-image): build multi-arch (amd64 + arm64) images
ci(ket-build-image): build multi-arch (amd64 + arm64) images
ci: build multi-arch docker image via native runners
ci(ket-build-image): build multi-arch (amd64 + arm64) images
ci(ket-build-image): build multi-arch (amd64 + arm64) images
ci(ket-build-image): build multi-arch (amd64 + arm64) images
ci(ket-build-image): build multi-arch (amd64 + arm64) images
ci(ket-build-image): build multi-arch (amd64 + arm64) images
ci(ket-build-image): build multi-arch (amd64 + arm64) images
ci(ket-build-image): build multi-arch (amd64 + arm64) images