Repository navigation
Description
Contributing guidelines
- I've read the contributing guidelines and wholeheartedly agree
I've found a bug and checked that ...
- ... the documentation does not mention anything about my problem
- ... there are no open or closed issues that are related to my problem
Description
I have been following the current documentation for how to use RUN --device within a Dockerfile but I am running into the following error when building the image with docker build -f Dockerfile .:
ERROR: failed to build: failed to solve: failed to load LLB: device nvidia.com/gpu=all is requested by the build but not allowed
If trying to allow the builder to access the device using docker build --allow=device=nvidia.com/gpu=all -f Dockerfile ., I get the following error:
ERROR: failed to build: failed to solve: granting entitlement device is not allowed by build daemon configuration
I can see in my output of docker buildx inspect (see below) that the GPU I have configured has not been automatically allowed to be used. Is this a bug? Or can the documentation for the GPU example of RUN --device be clarified to show how to allow the GPU?
docker buildx inspect default
Name: default
Driver: docker
Last Activity: 2025-07-04 08:34:31 +0000 UTC
Nodes:
Name: default
Endpoint: default
Status: running
BuildKit version: v0.23.2
Platforms: linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
Labels:
org.mobyproject.buildkit.worker.moby.host-gateway-ip: 172.17.0.1
Devices:
Name: nvidia.com/gpu=0
Automatically allowed: false
Name: nvidia.com/gpu=GPU-cf7e763a-628e-84d7-d146-62824b805c1f
Automatically allowed: false
Name: nvidia.com/gpu=all
Automatically allowed: false
GC Policy rule#0:
All: false
Filters: type==source.local,type==exec.cachemount,type==source.git.checkout
Keep Duration: 48h0m0s
Max Used Space: 4.375GiB
GC Policy rule#1:
All: false
Keep Duration: 1440h0m0s
Reserved Space: 31.66GiB
Max Used Space: 4.657GiB
Min Free Space: 15.83GiB
GC Policy rule#2:
All: false
Reserved Space: 31.66GiB
Max Used Space: 4.657GiB
Min Free Space: 15.83GiB
GC Policy rule#3:
All: true
Reserved Space: 31.66GiB
Max Used Space: 4.657GiB
Min Free Space: 15.83GiB
The CDI configuration was created by running: sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.json
Expected behaviour
For docker build -f Dockerfile . command to run with the Dockerfile without any errors.
Actual behaviour
Getting the following error:
ERROR: failed to build: failed to solve: failed to load LLB: device nvidia.com/gpu=all is requested by the build but not allowed
Buildx version
github.com/docker/buildx v0.25.0 faaea65
Docker info
Client: Docker Engine - Community
Version: 28.3.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.25.0
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.38.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 8
Server Version: 28.3.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Discovered Devices:
cdi: nvidia.com/gpu=0
cdi: nvidia.com/gpu=GPU-cf7e763a-628e-84d7-d146-62824b805c1f
cdi: nvidia.com/gpu=all
Swarm: inactive
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
runc version: v1.2.5-0-g59923ef
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-1031-aws
Operating System: Ubuntu 24.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 30.98GiB
Name: ip-10-200-3-199
ID: 89391cb8-ae75-4089-859f-472af634ad22
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
Builders list
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
default* docker
\_ default \_ default running v0.23.2 linux/amd64 (+3), linux/386
Configuration
# syntax=docker/dockerfile:1.17-labs
FROM nvidia/cuda:12.9.1-cudnn-runtime-ubuntu24.04
RUN --device=nvidia.com/gpu=all nvidia-smiBuild logs
Additional info
No response
Activity
crazy-max commentedon Jul 4, 2025
Oh good catch we forgot to document the
org.mobyproject.buildkit.device.autoallowannotation: https://github.com/moby/buildkit/blob/d5a40ad2a177b3d1141e83d4969831398abac83e/solver/llbsolver/cdidevices/manager.go#L20. I will open a PR on docs repo.So in your case you need to either set this annotation in your CDI specification similar to https://github.com/moby/buildkit/blob/d5a40ad2a177b3d1141e83d4969831398abac83e/solver/llbsolver/cdidevices/fixtures/vendor1-device.yaml#L8-L9
Or you can set the opt-in way using the
--allowflag: https://docs.docker.com/reference/cli/docker/buildx/build/#allow:(Needs to add documentation for this flag as well cc @ArthurFlag)
For example to allow any device:
oisinturbittcx1 commentedon Jul 4, 2025
Thank you very much for the comment, the help is much appreciated!
So I tried method 1 and got this error:
ERROR: failed to build: failed to solve: granting entitlement device is not allowed by build daemon configuration. That will mean that the default Docker builder instance can't be used here?Have also tried method 2, and while the output of
docker buildx inspect(see below) shows that the GPUs are now allowed, when actually running the build command it is still failing at theRUN --device=nvidia.com/gpu=all nvidia-smistep due to not picking up the Nvidia drivers:NVIDIA-SMI couldn't find libnvidia-ml.so library in your system. Please make sure that the NVIDIA Display Driver is properly installed and present in your system.Also, if anyone else stumbles across this, the CDI specification created by
nvidia-ctkisv0.5.0so the version field on the YAML spec needs to be manually changed to0.6.0to allow for theannotationsfield or else Docker won't load the specification.oisinturbittcx1 commentedon Jul 4, 2025
Also, just to confirm, running
docker run --rm --runtime=nvidia nvidia/cuda:12.9.1-base-ubuntu22.04 nvidia-smiis able to correctly pick up the GPU.oisinturbittcx1 commentedon Jul 4, 2025
I have tried
docker buildx create --use --bootstrap --name gpu-builder --buildkitd-flags '--allow-insecure-entitlement device'which resulted in the error below. Appears to not allow for the--allow-insecure-entitlement device'flag.crazy-max commentedon Jul 10, 2025
Thanks for your feedback!
Meaning
docker build -f Dockerfile --allow device .still error out withgranting entitlement device is not allowed by build daemon configuration? I will try to repro on my side.Indeed we should document that as well!
Ok that would mean that nvidia-ctk doesn't mount the driver correctly. Would mind sharing your YAML CDI spec and also the output of
nvidia-ctk --versionandnvidia-smicommands from your host please?Btw how did you install
nvidia-ctk? Did you follow https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#with-apt-ubuntu-debian? In my case I have done:In this case it doesn't use CDI but the NVIDIA container runtime.
Yes only
security.insecureandnetwork.hostare valid insecure entitlements.As this is working when using the nvidia container runtime we can try using a container builder that would make the GPU request and create the CDI spec automatically within this container.
For this we need a custom BuildKit image because current BuildKit release image is based on Alpine that doesn't support NVIDIA drivers. This is tracked in moby/buildkit#5944. For now I have pushed a temp image on Docker Hub:
crazymax/buildkit:v0.23.2-ubuntu-nvidia.Now you can create your
gpubuilderwith:Inspecting the builder:
We can see
nvidia.com/gpuvendor is detected as a device in the builder which means that drivers were detected.Optionally you can check if NVIDIA GPU devices are available in the container using
nvidia-smi:Now you should be able to build your Dockerfile with:
Note that you don't need an nvidia/cuda base image and can just use an ubuntu one as CDI mount the drivers.
During the build it will generate the CDI spec automatically. You should see smth like this in the build logs:
You can check the generated CDI specification within the container builder with the following command after the build:
oisinturbittcx1 commentedon Jul 16, 2025
Thank you so much for this! Using the
gpubuilderwith your image is working perfectly. I have been able to get this working with our workflow and it now allows us to compile CUDA kernels at image build time.pechersky commentedon Jul 18, 2025
I tried following the
gpubuildersuggestions (note that the comment and thus the commands below actually named the buildernvidia), but am still running into issues, with aFailed to initialize NVMLat the end, which then leads tonvidia-sminot found in later build attempts.crazy-max commentedon Jul 18, 2025
@oisinturbittcx1 Awesome! Would you mind sharing your YAML CDI spec and also the output of
nvidia-ctk --versionandnvidia-smicommands from your host please? So we can check this issue?@pechersky We had the same issues before on WSL but we solved this by making a device request for the container builder #3063. But since recent version on WSL it doesn't work anymore 😢.
It seems to be a limitation mounting dxcore within the container. Here we are installing the cuda drivers for
/dev/dxgdevice but it doesn't seem enough to make it works aslibdxcore.sois missing as you can see in the logs:If we look at
/usr/lib/wsl/libcontent we have on the host:I tried to find the package that installs these files in the WSL Ubuntu distribution (
dpkg -S) but seems there are none.Looking at the docs https://docs.nvidia.com/cuda//wsl-user-guide/index.html?utm_source=chatgpt.com#cuda-support-for-wsl-2, these files are available across all WSL distributions when drivers are installed on the host (Windows):
There are the meta packages as stated in "Option 2" but no linux drivers.
Mounting
/usr/lib/wslin BuildKit container seems to work https://github.com/docker/buildx/compare/master...crazy-max:buildx:driver-ctn-mount-wsl-lib?expand=1:Maybe this could be a workaround for WSL case but mounting from the host could be error prone if a remote context/builder is used. Could maybe be an opt-in or we could detect if we are using the default context and allow it in such case.
oisinturbittcx1 commentedon Jul 18, 2025
@crazy-max I got the above working on a machine that I no longer have access to unfortunately. But to do it, I followed the instructions here to install
nvidia-cktand get the following runningnvidia-ckt --version:Which creates the
/etc/cdi/nvidia.yamlcontents:Here is the log of the CDI spec being generated:
There appears to be a mismatch between the log line
INFO[0005] Generated CDI spec with version 0.8.0and thecdiVersion: 0.5.0in the actual YAML fileoisinturbittcx1 commentedon Jul 18, 2025
@crazy-max Working your instructions through to the end on this new machine results in a new error (see below). Appears to be that the Nvidia driver versions in
crazymax/buildkit:v0.23.2-ubuntu-nvidiais the desktop variant 575.64.03 while this new machine has the server variants 575.57.08:crazy-max commentedon Jul 18, 2025
@oisinturbittcx1 Thanks for your feedback
Looking at the logs:
Seems there is a mismatch with drivers and library version. It might be that we install latest client libraries during build but drivers on your host are too old and not compatible. @tonistiigi I wonder if we should pin client libraries to an older version to increase compat?
@pechersky Opened #3320 and seems to work on my side if you want to try.
crazy-max commentedon Jul 21, 2025
@oisinturbittcx1 Documentation published: https://docs.docker.com/build/building/cdi/
And fixes for this error here:
oisinturbittcx1 commentedon Jul 23, 2025
@crazy-max Thank you! The docs are great.
On the driver mismatch issue, I have been trying this issue on two different instances: a AWS EC2 g5.2xlarge and a LambdaLabs H100 server. Running the above setup on the H100 server works with no issues, however it is the EC2 instance that gives the driver mismatch error.
On the H100:
While on the A10G:
What is weird is that the H100 server has a driver version of
570.148.08while the A10G is on570.158.01. So it appears the H100 builder is pulling in the correct system driver while the A10G builder is pulling in an incorrect one. There is nothing obvious in the CDI for why it is looking for 570.172:oisinturbittcx1 commentedon Jul 28, 2025
@crazy-max got to the source of the driver mismatch. The A10G server I am on is using the the
nvidia-driver-XXX-serverdriver variants but thecrazymax/buildkit:v0.23.2-ubuntu-nvidiaimage requires thenvidia-driver-XXXdriver variants.crazy-max commentedon Jul 28, 2025
Thanks for your feedback I will check this on our side.
IAmIlliest commentedon Aug 13, 2025
FYI, I'm running into some similar issue as the OP, but haven't really got time to investigate it before giving the hardware back (this is a Jetson AGX Orin on Jetpack 6.2.1 and L4T 36.4.4).
Following these docs:
https://docs.docker.com/build/building/cdi/
Am I correct in my reading of those docs, and these steps are meant to be able to run with the default builder?
For what it's worth, on this host I can use CDI with docker run commands:
all devices
docker run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -Lspecific devices
Similarly, I can expose the GPU with the '--runtime=nvidia' flag for docker run as well so in general, the GPU's can be made accessible to containers for runtime. I'm needing it for build time though, similar to the OP.
In the building CDI docs (link at the top), I get down to the point of running the buildx commands with a dockerfile called CDI_dockerfile:
The first buildx command fails as expected:
The next step says to use the '--allow device' flag with the buildx command to fix the error above:
(The resultant error is the same here whether I have the allow annotation added in /etc/cdi/foo.yaml or not)
As you can see, the allow command is being disallowed by something in my daemon config (apparently). It's strange because the default builder can see the CDI devices. I tried explicitly enabling CDI in daemon.json along with setting the default runtime and then restarting docker.service, but the result is the same error. Here's a look at the daemon.json:
And a look at the default builder:
Same error results even with the allow annotations which you can see is working per the builder:
Any ideas if I'm doing something wrong here?
oisinturbittcx1 commentedon Aug 14, 2025
I have to do the following:
IAmIlliest commentedon Aug 15, 2025
Thanks for the clarification. I had gathered from this discussion that only worked with the new GPU builder. My confusion arose because the official docs seemed to imply it worked with the default builder....nevertheless, gpu-builder to the rescue.
crazy-max commentedon Aug 22, 2025
@IAmIlliest Yes this will be solved on next BuildKit release: #3295 (comment)