Skip to content

Add Concurrency Control to Matrix Builds #2581

Open

Description

@dennisoehme

Introduce an option to limit the number of concurrent builds when using Docker Buildx (Bake) to manage matrix builds. This feature would help users control system load and optimize resource utilization during complex build processes.

Description

Currently, Docker Buildx Bake allows users to define and run matrix builds, which can potentially execute a large number of targets in parallel. However, there is no built-in option to limit the concurrency of these builds. This can lead to excessive system load, especially on resource-constrained environments.

Proposed Solution

  1. Add a new option to the docker buildx bake command that allows users to specify the maximum number of concurrent build jobs. This option, tentatively named --max-parallel, would control the number of builds executed simultaneously.
  2. Add a concurrency field to the target definitions in the docker-bake.hcl file, enabling users to specify the maximum number of concurrent builds for each target group or individual target.
target "build" {
  inherits = ["_common"]
  name = "${PROJECT}-build"
  matrix = {
    "PROJECT" = [...]
  }
  concurrency = 2
}

Use Case(s)

  • External Download Limits: When Docker builds need to receive downloads from a third-party source that limits the number of parallel requests, controlling concurrency becomes essential. Without such control, exceeding the allowed number of parallel requests can result in build failures or throttling from the third-party service. By specifying concurrency limits in the build configuration, users can ensure compliance with these external constraints while optimizing build efficiency.
  • Differing Target Loads: Different build targets can produce varying loads on the system. For instance, a target that involves heavy compilation or resource-intensive operations might need to be limited to fewer concurrent builds compared to lighter targets. Specifying concurrency limits per target ensures that resource-intensive builds do not overload the system while allowing lighter builds to proceed in parallel, maximizing overall efficiency.

What do you think, guys? :-)


PS: I know we already have the max-parallelism option in buildkitd (moby/buildkit#1131), but this feature request describes a different purpose.

Activity

  1. added this to the v0.future milestone on Aug 9, 2024
  2. crazy-max commented on Aug 30, 2024

    @crazy-max
    Member

    Looks good but don't think this should be specific to matrix but overall invoked build targets so should be only set as flag or env imo.

  3. thompson-shaun commented on Sep 3, 2024

    @thompson-shaun
    Contributor

    Related? #359

  4. added
    kind/bugSomething isn't working
    and removed
    kind/bugSomething isn't working
    on Oct 8, 2024
  5. vishalnandagopal commented on Mar 12, 2025

    @vishalnandagopal

    External Download Limits: When Docker builds need to receive downloads from a third-party source that limits the number of parallel requests, controlling concurrency becomes essential. Without such control, exceeding the allowed number of parallel requests can result in build failures or throttling from the third-party service. By specifying concurrency limits in the build configuration, users can ensure compliance with these external constraints while optimizing build efficiency.

    This is an important use case.. If I have 5-6 JS projects, they are all built concurrently with docker bake. This leads to me being rate limited by NPM/yarn server or else just the network timing out.

    All targets are built successfully independently, and having 2-3 targets in default group is fine. But when i try to include more, it causes network errors.

    Sample errors, these are obviously not docker errors but yeah issue is due to too many concurrent builds.

    => => # info There appears to be trouble with your network connection. Retrying...
     => => # info There appears to be trouble with your network connection. Retrying...
     => => # info There appears to be trouble with your network connection. Retrying...
     => => # error An unexpected error occurred: "https://registry.npmjs.org/date-fns/-/date-fns-2.28.0.tgz: ESOCKETTIMEDOUT".
     => => # info If you think this is a bug, please open a bug report with the information provided in "/plugin/yarn-error.log".
     => => # info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
    ==> [plugin-a 5/7] RUN ["yarn", "install"]                                                                                                                                                              238.4s
     => => # warning jest > @jest/core > @jest/transform > babel-plugin-istanbul > test-exclude > glob@7.2.3: Glob versions prior to v9 are no longer supported
     => => # warning jest-environment-jsdom > jsdom > abab@2.0.6: Use your platform's native atob() and btoa() methods instead
     => => # warning jest-environment-jsdom > jsdom > data-urls > abab@2.0.6: Use your platform's native atob() and btoa() methods instead
     => => # warning jest-environment-jsdom > jsdom > domexception@4.0.0: Use your platform's native DOMException instead
     => => # [3/5] Fetching packages...
     => => # info There appears to be trouble with your network connection. Retrying`
    
  6. MalteMagnussen commented on May 27, 2025

    @MalteMagnussen

    Please add this. We are building over 150 images in one docker-bake.hcl file, and when they're all trying to push layers at the end, the entire thing freezes up.

    Please add the option to limit the builder to only work on X amount of things at a time.

    2000+ seconds of pushing layers
    [+] Building 2178.0s (883/961)                                             docker-container:docker-container
     => => pushing layers                                                                                1947.2s
     => [go_1_13_noble-rootless] exporting to image                                                      2091.7s
     => => pushing layers                                                                                1946.0s
     => [go_1_14_focal-rootless] exporting to image                                                      2089.8s
     => => pushing layers                                                                                1948.9s
     => [go_1_16_noble-rootless] exporting to image                                                      2089.6s
     => => pushing layers                                                                                1943.8s
     => [go_1_17_focal-rootless] exporting to image                                                      2089.1s
     => => pushing layers                                                                                1941.5s
     => [go_1_18_noble-rootless] exporting to image                                                      2087.3s
     => => pushing layers                                                                                1937.3s
     => [go_1_19_jammy-rootless] exporting to image                                                      2079.5s
     => => pushing layers                                                                                1936.8s
     => [bazel_8_2_1_noble-rootless] exporting to image                                                  2058.5s
     => => pushing layers                                                                                1955.2s
     => [go_1_24_focal-rootless] exporting to image                                                      2051.6s
     => => pushing layers                                                                                1938.2s
     => [go_1_23_noble-rootless] exporting to image                                                      2051.5s
     => => pushing layers                                                                                1940.2s
     => [go_1_15_jammy-rootless] exporting to image                                                      2051.3s
     => => pushing layers                                                                                1938.8s
     => [go_1_13_jammy-rootless] exporting to image                                                      2050.8s
     => => pushing layers                                                                                1940.7s
     => [go_1_18_focal-rootless] exporting to image                                                      2050.0s
     => => pushing layers                                                                                1935.8s
     => [java-8-11-gradle_7_4_2_focal-rootless] exporting to image                                       2050.0s
     => => pushing layers                                                                                1947.7s
     => [go_1_14_jammy-rootless] exporting to image                                                      2044.3s
     => => pushing layers                                                                                1937.7s
     => [python_runtime_3_12_jammy-rootless] exporting to image                                          2043.9s
     => => pushing layers                                                                                1955.5s
     => [python_runtime_3_10_noble-rootless] exporting to image                                          2041.6s
     => => pushing layers                                                                                1955.3s
     => [python_runtime_3_8_noble-rootless] exporting to image                                           2040.1s
     => => pushing layers                                                                                1955.3s
     => [python_runtime_3_8_jammy-rootless] exporting to image                                           2040.1s
     => => pushing layers                                                                                1955.2s
     => [python_runtime_3_10_jammy-rootless] exporting to image                                          2039.9s
     => => pushing layers                                                                                1954.4s
     => [hugo_0_136_5_jammy-rootless] exporting to image                                                 2018.0s
     => => pushing layers                                                                                1955.5s
     => [hugo_0_128_2_jammy-rootless] exporting to image                                                 2017.6s
     => => pushing layers                                                                                1955.2s
     => [node_18_jammy-rootless] exporting to image                                                      1996.4s
     => => pushing layers                                                                                1945.3s
     => [node_18_noble-rootless] exporting to image                                                      1991.3s
     => => pushing layers                                                                                1943.3s
     => [java-17-maven_3_8_8_focal-rootless] exporting to image                                          1991.2s
     => => pushing layers                                                                                1942.0s
     => [go_1_23_jammy-rootless] exporting to image                                                      1985.7s
     => => pushing layers                                                                                1930.6s
     => [java-17-gradle_8_0_1_focal-rootless] exporting to image                                         1984.5s
     => => pushing layers                                                                                1941.3s
     => [java-17-gradle_7_6_focal-rootless] exporting to image                                           1984.5s
     => => pushing layers                                                                                1941.2s
     => [go_1_13_focal-rootless] exporting to image                                                      1983.9s
     => => pushing layers                                                                                1928.4s
     => [go_1_18_jammy-rootless] exporting to image                                                      1983.3s
     => => pushing layers                                                                                1923.9s
     => [java-17-gradle_7_4_2_focal-rootless] exporting to image                                         1982.5s
     => => pushing layers                                                                                1942.2s
     => [busybox-webserver_noble-rootless] exporting to image                                            1982.4s
     => => pushing layers                                                                                1954.2s
     => [docker-buildkit_jammy-rootless] exporting to image                                              1951.4s
     => => pushing layers                                                                                1936.9s
     => [node_20_noble-rootless] exporting to image                                                      1945.1s
     => => pushing layers                                                                                1929.6s
     => [docker-buildkit_noble-rootless] exporting to image                                              1925.8s
     => => pushing layers                                                                                1916.6s
     => [node_22_jammy-rootless] exporting to image                                                      1924.5s
     => => pushing layers                                                                                1915.0s
     => [node_20_jammy-rootless] exporting to image                                                      1907.7s
     => => pushing layers                                                                                1902.1s
     => [bazel_8_2_1_jammy-rootless] exporting to image                                                  1901.4s
     => => pushing layers                                                                                1899.4s
    
  7. crazy-max commented on Jul 24, 2025

    @crazy-max
    Member

    Please add this. We are building over 150 images in one docker-bake.hcl file, and when they're all trying to push layers at the end, the entire thing freezes up.

    Please add the option to limit the builder to only work on X amount of things at a time.

    @MalteMagnussen As you're building with the docker-container driver you can already limit parallelism in the builder directly: https://docs.docker.com/build/buildkit/configure/#max-parallelism

    Create this TOML file:

    [worker.oci]
      max-parallelism = 4
    $ docker buildx create --use \
      --name mybuilder \
      --driver docker-container \
      --buildkitd-config /tmp/buildkitd.toml
    

    You can place it in default locations: https://docs.docker.com/reference/cli/docker/buildx/create/#buildkitd-config

    Like ~/.docker/buildx/buildkitd.default.toml so you don't need to pass it everytime you invoke buildx create:

    $ docker buildx create --use \
      --name mybuilder \
      --driver docker-container
    

    I will look if we can have a buildkitd flag to set max parallelism so you don't need to create a config file just for this.

  8. MalteMagnussen commented on Jul 25, 2025

    @MalteMagnussen

    @crazy-max Thanks!! I really appreciate it. I'll add that to our docs.

  9. MalteMagnussen commented on Aug 7, 2025

    @MalteMagnussen

    @crazy-max Ended up adding this to our repo.

    buildkitd.toml

    # The TOML file used to configure the buildkitd daemon settings 
    # has a short list of global settings followed by a series of sections for specific areas of daemon configuration.
    #
    # The file path is /etc/buildkit/buildkitd.toml for rootful mode, 
    # ~/.config/buildkit/buildkitd.toml for rootless mode.
    #
    # https://docs.docker.com/build/buildkit/toml-configuration/
    
    [worker.oci]
      # When too many workers are running, it can actually slow down the build process, or freeze the machine.
      # So many images can be open at once, that your machine runs out of memory, and starts using the swap file.
      max-parallelism = 6

    Makefile

    bake-builder-local:
    	@echo "Creating local builder..."
    	docker buildx create --use --driver docker-container --name docker-container --buildkitd-config ./buildkitd.toml
    	docker buildx inspect
  10. crazy-max commented on Aug 7, 2025

    @crazy-max
    Member

    bake-builder-local:
    @echo "Creating local builder..."
    docker buildx create --use --driver docker-container --name docker-container --buildkitd-config ./buildkitd.toml
    docker buildx inspect

    In next BuildKit version (via moby/buildkit#6105), the config will not be needed so you can just do:

    docker buildx create --use --driver docker-container --name docker-container --buildkitd-flags '--oci-max-parallelism=6'
    
  11. BtbN commented on Aug 15, 2025

    @BtbN

    I'm also desperately looking for a way to control concurrency/max-parallelism on a "per build" level.
    I.e. I'd like to do "docker buildx build --jobs=4" or something like that, basically the same as the issue is asking for, but without bake in the mix. At least if I understand the proposal right.

    The reason is that I have a chain of highly parallel docker images that depend on one another. I'm currently forced to use a docker-container builder for them, which causes a lot of pain because I have to jump through many hoops to load previously built stages into it, and get them out of there, while maintaining proper cache.

    So having a way to limit the number of parallel steps running on the default docker builder would make a lot of things a lot better.

    Right now I'm forced to either deal with constantly copying images around, wasting tons of time, and having them stored multiple times, wasting tons of space, or deal with buildx build spawning dozens of parallel steps, exhausting all available resources of the system, and eventually crashing something.

    If I understand things right, the first thing that'd need to happen for this would be to add a way to pass a per-build max-parallelism value to https://github.com/moby/buildkit and once that is in wire it up here, so that "buildx build" can pass something like a "--jobs" setting to buildkit?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions