Skip to content

'buildx imagetools create' broken since v0.31.1 when building multi-platform images on github across multiple runners #3708

Closed

Description

@yubiuser

Contributing guidelines

I've found a bug and checked that ...

  • ... the documentation does not mention anything about my problem
  • ... there are no open or closed issues that are related to my problem

Description

We've been using a github workflow which basically follows the steps outlined in

https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners

Our workflow file is here.

We noticed, the 'merge' step failed since 6th of February, which was the day docker/setup-buildx-action started to deploy buildx version v0.31.1.
The logs always error at the docker buildx imagetools create step with

#1 [internal] pushing ghcr.io/pi-hole/***
#1 24.69 pushing sha256:1ac73ae4246c36835675145362d146a4bd2af7bd518de3e2bda44cf62116b55d to ghcr.io/pi-hole/***:nightly
#1 DONE 25.1s
ERROR: ghcr.io/pi-hole/***@sha256:f4a9f4daa0088ae6b74c45149690f9c31420122ad9079c231df81b2e31629010: not found
Error: Process completed with exit code 1.

We've manually reverted to buildx v0.31.0 and it immediately started to work again.
Here is a github action run which failed
https://github.com/pi-hole/docker-pi-hole/actions/runs/22030313470/job/63654289108

and this one is right after the version was set to v0.31.0
https://github.com/pi-hole/docker-pi-hole/actions/runs/22039970323/job/63679202890

As v0.31.1 only contained a single PR, it's probably due to #3632

Expected behaviour

Following the official guide for creating multi-arch images across multiple runners should not fail at step docker buildx imagetools create

Actual behaviour

See above. Workflows fail at the docker buildx imagetools create step.

Buildx version

affected are all versions > 0.31.0

Docker info


Builders list

Not sure how to obtain at github action runners.

Configuration

This is the Dockerfile we build: https://github.com/pi-hole/docker-pi-hole/blob/master/src/Dockerfile
And this is the corresponding workflow: https://github.com/pi-hole/docker-pi-hole/blob/master/.github/workflows/build-and-publish.yml

Build logs


Additional info

No response

Activity

  1. thaJeztah commented on Mar 7, 2026

    @thaJeztah
    Member
  2. crazy-max commented on Mar 8, 2026

    @crazy-max
    Member

    Thanks for your report.

    What changed in v0.31.1 is imagetools create copy behavior from #3623: it now copies only the selected descriptor(s), instead of relying on broader copy side effects. Your workflow runs imagetools create twice and reuses all metadata tags (docker hub + GHCR) for both runs; with this pattern, the second run can reference ghcr.io/...@sha256:... digests that are not present in GHCR yet, which matches the not found error you see.

    So this is reproducible with that workflow shape after v0.31.1. The immediate workaround is to split tags per registry (Docker Hub tags for the Docker Hub create call, GHCR tags for the GHCR call), or otherwise ensure each create call uses source digests from the same registry it is targeting. Something similar to: https://github.com/crazy-max/docker-healthchecks/blob/eef77d758b6c8b06220adc3e4b4f11da1eca6a89/.github/workflows/build.yml#L188-L191

    Also, if you want to simplify this flow, consider switching to our GitHub Builder. This keeps multi-platform image publishing to Docker Hub and GHCR while simplifying the repository-specific GitHub Actions logic. Also reduces CI workflow complexity and maintenance burden for multi-platform builds. Compared to our "Distribute build across multiple runners" docs, this removes the need for per-platform matrix orchestration, digest artifact handoff, and a separate manifest merge job.

    If you're interested I pushed these changes on a branch: crazy-max/docker-pi-hole@7195273. There is just this build-arg PIHOLE_DOCKER_TAG=${{ steps.meta.outputs.version }} that would not work because metadata outputs are not exposed here but that probably something we could improve in our reusable workflows using handlebars templates.

  3. yubiuser commented on Mar 9, 2026

    @yubiuser
    Author

    Thanks for your comprehensive answer. You really went the extra mile!

    I now see where and why we have the issue here. I can implement the workaround but your GitHub Builder looks way more interesting! It would free us from a huge maintenance burden, because handling and orchestrating those distributed builds was not fun. I'm looking forward for the implementation of

    metadata outputs are not exposed

    and then I'll switch.

  4. tonistiigi commented on Mar 10, 2026

    @tonistiigi
    Member

    Some more summary for this issue in case anyone else hits something similar.

    imagetools create has 3 possible ways it can create target image:

    1. make a carbon-copy if single arg and no --platform
    2. make a merged copy if multiple sources
    3. make a filtered copy if --platform filter set.

    The case that seems to be hit in here is merged copy where sources are image indexes. In this case the source image index is discarded and replaced with the merged index. Before v0.31.1 (possibly after --platform was added and before v0.31.1) we used to have a bug where we still accidentally copied the source index (and descriptors for unused platforms) as well, even though they remained unreferenced by the target image. This was fixed in v0.31.1 .

    If there is a valid use case to access the source index but leave it untagged, we could consider an extra option that would allow push-by-digest when creating the target image.

    @yubiuser If this makes sense then I think we can close this issue. If you think push-by-digest could be useful, you can open a new issue to track that.

  5. yubiuser commented on Mar 12, 2026

    @yubiuser
    Author

    I think we can close the issue. You have been more than helpful.
    Once we switch to the GitHub Builder, we don't have a need for push-by-digest. This was only ever a mean to get things done, nothing we loved to deal with.

  6. crazy-max commented on Mar 13, 2026

    @crazy-max
    Member
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions