Repository navigation
Open
Labels
area/bakekind/enhancementNew feature or requestNew feature or requeststatus/needs-maintainer-decision
Milestone
Description
Introduce an option to limit the number of concurrent builds when using Docker Buildx (Bake) to manage matrix builds. This feature would help users control system load and optimize resource utilization during complex build processes.
Description
Currently, Docker Buildx Bake allows users to define and run matrix builds, which can potentially execute a large number of targets in parallel. However, there is no built-in option to limit the concurrency of these builds. This can lead to excessive system load, especially on resource-constrained environments.
Proposed Solution
- Add a new option to the
docker buildx bakecommand that allows users to specify the maximum number of concurrent build jobs. This option, tentatively named--max-parallel, would control the number of builds executed simultaneously. - Add a
concurrencyfield to the target definitions in thedocker-bake.hclfile, enabling users to specify the maximum number of concurrent builds for each target group or individual target.
target "build" {
inherits = ["_common"]
name = "${PROJECT}-build"
matrix = {
"PROJECT" = [...]
}
concurrency = 2
}Use Case(s)
- External Download Limits: When Docker builds need to receive downloads from a third-party source that limits the number of parallel requests, controlling concurrency becomes essential. Without such control, exceeding the allowed number of parallel requests can result in build failures or throttling from the third-party service. By specifying concurrency limits in the build configuration, users can ensure compliance with these external constraints while optimizing build efficiency.
- Differing Target Loads: Different build targets can produce varying loads on the system. For instance, a target that involves heavy compilation or resource-intensive operations might need to be limited to fewer concurrent builds compared to lighter targets. Specifying concurrency limits per target ensures that resource-intensive builds do not overload the system while allowing lighter builds to proceed in parallel, maximizing overall efficiency.
What do you think, guys? :-)
PS: I know we already have the max-parallelism option in buildkitd (moby/buildkit#1131), but this feature request describes a different purpose.
Reacted by Vishal N, Thomas Hanser, Malte Hviid-Magnussen, Marc Porciuncula, Sergio Bergamini, Sam Rodkey, Christian Menard, Brennan Kinney, Rembert, mbe9 and 3 more
Metadata
Metadata
Assignees
Labels
area/bakekind/enhancementNew feature or requestNew feature or requeststatus/needs-maintainer-decision
Activity
crazy-max commentedon Aug 30, 2024
Looks good but don't think this should be specific to
matrixbut overall invoked build targets so should be only set as flag or env imo.thompson-shaun commentedon Sep 3, 2024
Related? #359
vishalnandagopal commentedon Mar 12, 2025
This is an important use case.. If I have 5-6 JS projects, they are all built concurrently with docker bake. This leads to me being rate limited by NPM/yarn server or else just the network timing out.
All targets are built successfully independently, and having 2-3 targets in
defaultgroup is fine. But when i try to include more, it causes network errors.Sample errors, these are obviously not docker errors but yeah issue is due to too many concurrent builds.
MalteMagnussen commentedon May 27, 2025
Please add this. We are building over 150 images in one docker-bake.hcl file, and when they're all trying to push layers at the end, the entire thing freezes up.
Please add the option to limit the builder to only work on X amount of things at a time.
2000+ seconds of pushing layers
crazy-max commentedon Jul 24, 2025
@MalteMagnussen As you're building with the
docker-containerdriver you can already limit parallelism in the builder directly: https://docs.docker.com/build/buildkit/configure/#max-parallelismCreate this TOML file:
You can place it in default locations: https://docs.docker.com/reference/cli/docker/buildx/create/#buildkitd-config
Like
~/.docker/buildx/buildkitd.default.tomlso you don't need to pass it everytime you invokebuildx create:I will look if we can have a buildkitd flag to set max parallelism so you don't need to create a config file just for this.
MalteMagnussen commentedon Jul 25, 2025
@crazy-max Thanks!! I really appreciate it. I'll add that to our docs.
MalteMagnussen commentedon Aug 7, 2025
@crazy-max Ended up adding this to our repo.
buildkitd.tomlMakefilecrazy-max commentedon Aug 7, 2025
In next BuildKit version (via moby/buildkit#6105), the config will not be needed so you can just do:
BtbN commentedon Aug 15, 2025
I'm also desperately looking for a way to control concurrency/max-parallelism on a "per build" level.
I.e. I'd like to do "docker buildx build --jobs=4" or something like that, basically the same as the issue is asking for, but without bake in the mix. At least if I understand the proposal right.
The reason is that I have a chain of highly parallel docker images that depend on one another. I'm currently forced to use a docker-container builder for them, which causes a lot of pain because I have to jump through many hoops to load previously built stages into it, and get them out of there, while maintaining proper cache.
So having a way to limit the number of parallel steps running on the default docker builder would make a lot of things a lot better.
Right now I'm forced to either deal with constantly copying images around, wasting tons of time, and having them stored multiple times, wasting tons of space, or deal with buildx build spawning dozens of parallel steps, exhausting all available resources of the system, and eventually crashing something.
If I understand things right, the first thing that'd need to happen for this would be to add a way to pass a per-build max-parallelism value to https://github.com/moby/buildkit and once that is in wire it up here, so that "buildx build" can pass something like a "--jobs" setting to buildkit?