Skip to content

Error when using RUN --device #3295

Closed
Closed

Description

@oisinturbittcx1

Contributing guidelines

I've found a bug and checked that ...

  • ... the documentation does not mention anything about my problem
  • ... there are no open or closed issues that are related to my problem

Description

I have been following the current documentation for how to use RUN --device within a Dockerfile but I am running into the following error when building the image with docker build -f Dockerfile .:

ERROR: failed to build: failed to solve: failed to load LLB: device nvidia.com/gpu=all is requested by the build but not allowed

If trying to allow the builder to access the device using docker build --allow=device=nvidia.com/gpu=all -f Dockerfile ., I get the following error:

ERROR: failed to build: failed to solve: granting entitlement device is not allowed by build daemon configuration

I can see in my output of docker buildx inspect (see below) that the GPU I have configured has not been automatically allowed to be used. Is this a bug? Or can the documentation for the GPU example of RUN --device be clarified to show how to allow the GPU?

docker buildx inspect default
Name:          default
Driver:        docker
Last Activity: 2025-07-04 08:34:31 +0000 UTC

Nodes:
Name:             default
Endpoint:         default
Status:           running
BuildKit version: v0.23.2
Platforms:        linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
Labels:
 org.mobyproject.buildkit.worker.moby.host-gateway-ip: 172.17.0.1
Devices:
 Name:                  nvidia.com/gpu=0
 Automatically allowed: false
 Name:                  nvidia.com/gpu=GPU-cf7e763a-628e-84d7-d146-62824b805c1f
 Automatically allowed: false
 Name:                  nvidia.com/gpu=all
 Automatically allowed: false
GC Policy rule#0:
 All:            false
 Filters:        type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration:  48h0m0s
 Max Used Space: 4.375GiB
GC Policy rule#1:
 All:            false
 Keep Duration:  1440h0m0s
 Reserved Space: 31.66GiB
 Max Used Space: 4.657GiB
 Min Free Space: 15.83GiB
GC Policy rule#2:
 All:            false
 Reserved Space: 31.66GiB
 Max Used Space: 4.657GiB
 Min Free Space: 15.83GiB
GC Policy rule#3:
 All:            true
 Reserved Space: 31.66GiB
 Max Used Space: 4.657GiB
 Min Free Space: 15.83GiB

The CDI configuration was created by running: sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.json

Expected behaviour

For docker build -f Dockerfile . command to run with the Dockerfile without any errors.

Actual behaviour

Getting the following error:

ERROR: failed to build: failed to solve: failed to load LLB: device nvidia.com/gpu=all is requested by the build but not allowed

Buildx version

github.com/docker/buildx v0.25.0 faaea65

Docker info

Client: Docker Engine - Community
 Version:    28.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.25.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.38.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 8
 Server Version: 28.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Discovered Devices:
  cdi: nvidia.com/gpu=0
  cdi: nvidia.com/gpu=GPU-cf7e763a-628e-84d7-d146-62824b805c1f
  cdi: nvidia.com/gpu=all
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-1031-aws
 Operating System: Ubuntu 24.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.98GiB
 Name: ip-10-200-3-199
 ID: 89391cb8-ae75-4089-859f-472af634ad22
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
default*      docker                                 
 \_ default    \_ default       running   v0.23.2    linux/amd64 (+3), linux/386

Configuration

# syntax=docker/dockerfile:1.17-labs
FROM nvidia/cuda:12.9.1-cudnn-runtime-ubuntu24.04

RUN --device=nvidia.com/gpu=all nvidia-smi

Build logs


Additional info

No response

Activity

  1. crazy-max commented on Jul 4, 2025

    @crazy-max
    Member

    I can see in my output of docker buildx inspect (see below) that the GPU I have configured has not been automatically allowed to be used. Is this a bug?

    Oh good catch we forgot to document the org.mobyproject.buildkit.device.autoallow annotation: https://github.com/moby/buildkit/blob/d5a40ad2a177b3d1141e83d4969831398abac83e/solver/llbsolver/cdidevices/manager.go#L20. I will open a PR on docs repo.

    So in your case you need to either set this annotation in your CDI specification similar to https://github.com/moby/buildkit/blob/d5a40ad2a177b3d1141e83d4969831398abac83e/solver/llbsolver/cdidevices/fixtures/vendor1-device.yaml#L8-L9

    Or you can set the opt-in way using the --allow flag: https://docs.docker.com/reference/cli/docker/buildx/build/#allow:

    (Needs to add documentation for this flag as well cc @ArthurFlag)

    --allow device grants access to any device.
    --allow device=kind|name grants access to specific device.
    --allow device=kind|name,alias=kind|name allows mapping kind to a specific device or one device to another. Alias is the name requested by the build and device is the actual device that is being enabled.
    

    For example to allow any device:

    docker build -f Dockerfile --allow device .
    
  2. oisinturbittcx1 commented on Jul 4, 2025

    @oisinturbittcx1
    Author

    Thank you very much for the comment, the help is much appreciated!

    So I tried method 1 and got this error: ERROR: failed to build: failed to solve: granting entitlement device is not allowed by build daemon configuration. That will mean that the default Docker builder instance can't be used here?

    Have also tried method 2, and while the output of docker buildx inspect (see below) shows that the GPUs are now allowed, when actually running the build command it is still failing at the RUN --device=nvidia.com/gpu=all nvidia-smi step due to not picking up the Nvidia drivers: NVIDIA-SMI couldn't find libnvidia-ml.so library in your system. Please make sure that the NVIDIA Display Driver is properly installed and present in your system.

    Also, if anyone else stumbles across this, the CDI specification created by nvidia-ctk is v0.5.0 so the version field on the YAML spec needs to be manually changed to 0.6.0 to allow for the annotations field or else Docker won't load the specification.

    Name:          default
    Driver:        docker
    Last Activity: 2025-07-04 15:05:07 +0000 UTC
    
    Nodes:
    Name:             default
    Endpoint:         default
    Status:           running
    BuildKit version: v0.23.2
    Platforms:        linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
    Labels:
     org.mobyproject.buildkit.worker.moby.host-gateway-ip: 172.17.0.1
    Devices:
     Name:                  nvidia.com/gpu=0
     Automatically allowed: true
     Annotations:
                            org.mobyproject.buildkit.device.autoallow: true
     Name:                  nvidia.com/gpu=GPU-cf7e763a-628e-84d7-d146-62824b805c1f
     Automatically allowed: true
     Annotations:
                            org.mobyproject.buildkit.device.autoallow: true
     Name:                  nvidia.com/gpu=all
     Automatically allowed: true
     Annotations:
      org.mobyproject.buildkit.device.autoallow: true
    GC Policy rule#0:
     All:            false
     Filters:        type==source.local,type==exec.cachemount,type==source.git.checkout
     Keep Duration:  48h0m0s
     Max Used Space: 4.375GiB
    GC Policy rule#1:
     All:            false
     Keep Duration:  1440h0m0s
     Reserved Space: 31.66GiB
     Max Used Space: 4.657GiB
     Min Free Space: 15.83GiB
    GC Policy rule#2:
     All:            false
     Reserved Space: 31.66GiB
     Max Used Space: 4.657GiB
     Min Free Space: 15.83GiB
    GC Policy rule#3:
     All:            true
     Reserved Space: 31.66GiB
     Max Used Space: 4.657GiB
     Min Free Space: 15.83GiB
    
  3. oisinturbittcx1 commented on Jul 4, 2025

    @oisinturbittcx1
    Author

    Also, just to confirm, running docker run --rm --runtime=nvidia nvidia/cuda:12.9.1-base-ubuntu22.04 nvidia-smi is able to correctly pick up the GPU.

  4. oisinturbittcx1 commented on Jul 4, 2025

    @oisinturbittcx1
    Author

    I have tried docker buildx create --use --bootstrap --name gpu-builder --buildkitd-flags '--allow-insecure-entitlement device' which resulted in the error below. Appears to not allow for the --allow-insecure-entitlement device' flag.

     > [internal] booting buildkit:                                                        
    18.98 time="2025-07-04T15:37:32Z" level=info msg="found worker \"jdpy0jn4glj9yrk5rkentime="2025-07-04T15:37:32Z" level=info msg="found worker \"jdpy0jn4glj9yrk5rkens8ntime="2025-07-04T15:37:32Z" level=info msg="found worker \"jdpy0jn4glj9yrk5rkens8nmm\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:02c07f72cf11 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/386]"
    18.98 time="2025-07-04T15:37:32Z" level=warning msg="currently, only the default worker can be used."
    18.98 (buildkitd: invalid entitletime="2025-07-04T15:37:32Z" level=warning msg="currently, only the default worker can be used."
    18.98 time="2025-07-04T15:37:32Z" level=warning msg="currently, only the default worker can be used."
    18.98   /src/cmd/buildkitd/main.go:368
    18.98 # /src/cmd/buildk /src/cmd/buildkitd/main.go:368
    18.98 github.com/urfave/cli.(*App).Run
    18.98 github.com/urfave/cli.(*App).Run
    18.98 . /srchub.lumenfield.work/urfave/cli.(*App).Run
    18.98   /usr/local/go/src/runtime/proc. /usr/local/g    /usr/local/go/src/runti/usr/local/go/stime="2025-07-0time="2025-07-04T15:37:33Z" level=info msg="time="2025-07-04T15:37:33Z" level=info msg="auto snapshotter: using overlayfs"
    18.98 time="2025-07-04T15:37:33Z" level=warning msg="skipping containerd worker, as \"/rtime="2025-07-04T15:37:33Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
    18.98 dtime="2025-07-04T15:37:33Z" level=info msg="found 1 workers, default=\"jdpy0jn4glj9yrk5rkens8nmm\""
    18.98 `time="2025-07-04T15:37:33Z" level=warning msg="currently, only the default worker can be used."
    18.98 (buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/main.go:368
    18.98 #github.com/urtime="2025-07-04T15:37:33Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
    18.98 buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/main.go:buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/mainbuildkitd: invalid entitlement : device
    18.98 github.com/urfavhub.lumenfield.work/urfave/cli.HandleActihub.lumenfield.work/urfave/cli.HandleAction
    18.98   /src/vendor/github.com/urfave/cli/app.go:286
    18.98   /src/vendor/github.com/urfave/cl        /src/vendor/github.com/urfave/cli/app.go:286
    18.98 runtime.maruntime.main
    18.98 ' /usr/localruntime.main
    18.98   /usr/local/go/src/runtime/asm_amd64.s:  /usr/local/go/  /usr/local/go/src/runtime/asm_amd64.s:1700
    18.98 time="2025-07-04T15:37:33Z" level=info msg="found worker \"jdpy0jn4glj9yrk5rkentime="2025-07-04T15:37:33Z" level=info msg="found worker \"jdpy0jn4glj9yrk5rkens8ntime="2025-07-04T15:37:33Z" level=info msg="found 1 workers, default=\"jdpy0jn4glj9yrk5rkens8nmm\""
    18.98 `time="2025-07-04T15:37:33Z" level=warning msg="currently, only the default worker can be used."
    18.98 (buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/main.go:368
    18.98 #github.com/urfave/cli.HandleAction
    18.98 . /src/vendor/github.com/urfave/cli/app.go:524
    18.98 !github.com/urfave/cli.(*App).Run
    18.98 . /src/vendor/time="2025-07-04T15:37:33Z" level=info msg="found 1 workers, default=\"jdpy0jn4glj9yrk5rkens8nmm\""
    18.98 `time="2025-07-04T15:37time="2025-07-04T15:37:33Z" level=info msg="found 1 workers, default=\"jdpy0jn4glj9yrk5rkens8nmm\""
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/main.go:368
    18.98 #github.com/urfave/cli.HandleActimain.main.func3
    18.98   /src/cmd/buildkmain.main.func3
    18.98   /src/vendor/github.com/urfave/c /src/vendor/github.com/urfave/cli/      /src/vendor/github.com/urfave/cli/app.go:524
    18.98 main.main
    18.98   /src/cmd/buildmain.main
    18.98   /src/cmd/buildkitd/main.go:main.main
    18.98   /usr/local/go/src/runtime/proc. /usr/local/g    /usr/local/go/src/runtime/proc.go:283
    18.98 time="2025-07-0time="2025-07-04T15:37:34Z" level=info msg="time="2025-07-04T15:37:34Z" level=info msg="auto snapshotter: using overlayfs"
    18.98 time="2025-07-04T15:37:34Z" level=warning msg="failed to enable cgroup controller time="2025-07-04T15:37:34Z" level=warning msg="failed to enable cgroup controller \"cpu\": write /sys/fs/cgroup/cgroup.subtree_control: device or resource busy"
    18.98 time="2025-07-04T15:37:34Z" level=warning msg="failed to enable cgroup controller \"memory\": write /sys/fs/cgroup/cgroup.subtree_control: device or resource bustime="2025-07-04T15:37:34Z" level=warning msg="failed to enable cgroup controller \"memory\": write /sys/fs/cgroup/cgroup.subtree_control: device or resource butime="2025-07-04T15:37:34Z" level=warning msg="failed to enable cgroup controller \"memory\": write /sys/fs/cgroup/cgroup.subtree_control: device or resource busy"
    18.98 time="2025-07-04T15:37:34Z" level=warning msg="failed to enable cgroup controller \"rdma\": write /sys/fs/cgroup/cgroup.subtree_control: device or resource busy"
    18.98 time="2025-07-04T15:37:34Z" level=warning msg="failed to enable cgroup controller \"rdma\": write /sys/fs/cgroup/cgroup.subtree_control: device or resource busy"
    18.98 time="2025-07-04T15:37:34Z" level=warning msg="failed to enable cgroup controller \"rdma\": write /sys/fs/cgroup/cgroup.subtree_control: device or resource busy"
    18.98 time="2025-07-04T15:37:34Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
    18.98 dtime="2025-07-04T15:37:34Z"time="2025-07-04T15:37:34Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
    18.98 dtime="2025-07-04T15:37:34Z" level=info msg="found 1 workers, default=\"jdpy0jn4glj9yrk5rkens8nmm\""
    18.98 `time="2025-07-04T15:37:34Z" level=warning msg="currently, only the default worker can be used."
    18.98 (buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98 time="2025-07-04T15:37:34Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
    18.98 buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98 buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98 buildkitd: invalid entit  /src/vendor/github.com/urfave/c /src/vendor/github.com/urfave/cli/      /src/vendor/github.com/urfave/cli/app.go:524
    18.98 main.main
    18.98   /src/cmd/buildmain.main
    18.98   /src/cmd/buildkitd/main.go:main.main
    18.98   /usr/local/go/src/runtime/proc. /usr/local/g    /usr/local/go/src/runtime/proc.go:283
    18.98 time="2025-07-0time="2025-07-04T15:37:35Z" level=info msg="time="2025-07-04T15:37:35Z" level=info msg="auto snapshotter: using overlayfs"
    18.98 time="2025-07-04T15:37:35Z" level=warning msg="skipping containerd worker, as \"/rtime="2025-07-04T15:37:35Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
    18.98 dtime="2025-07-04T15:37:35Z" level=info msg="found 1 workers, default=\"jdpy0jn4glj9yrk5rkens8nmm\""
    18.98 `time="2025-07-04T15:37:35Z" level=warning msg="currently, only the default worker can be used."
    18.98 (buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/main.go:368
    18.98 #github.com/urtime="2025-07-04T15:37:35Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
    18.98 buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/main.go:buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/mainbuildkitd: invalid entitlement : device
    18.98 github.com/urfavhub.lumenfield.work/urfave/cli.HandleActihub.lumenfield.work/urfave/cli.HandleAction
    18.98   /src/vendor/github.com/urfave/cli/app.go:286
    18.98   /src/vendor/github.com/urfave/cl        /src/vendor/github.com/urfave/cli/app.go:286
    18.98 runtime.maruntime.main
    18.98 ' /usr/localruntime.main
    18.98   /usr/local/go/src/runtime/asm_amd64.s:  /usr/local/go/  /usr/local/go/src/runtime/asm_amd64.s:1700
    18.98 time="2025-07-04T15:37:36Z" level=info msg="found worker \"jdpy0jn4glj9yrk5rkentime="2025-07-04T15:37:36Z" level=info msg="found worker \"jdpy0jn4glj9yrk5rkens8ntime="2025-07-04T15:37:36Z" level=info msg="found worker \"jdpy0jn4glj9yrk5rkens8nmm\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:02c07f72cf11 org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/386]"
    18.98 time="2025-07-04T15:37:36Z" level=warning msg="currently, only the default worker can be used."
    18.98 (buildkitd: invalid entitletime="2025-07-04T15:37:36Z" level=warning msg="currently, only the default worker can be used."
    18.98 time="2025-07-04T15:37:36Z" level=warning msg="currently, only the default worker can be used."
    18.98   /src/cmd/buildkitd/main.go:368
    18.98 # /src/cmd/buildk /src/cmd/buildkitd/main.go:368
    18.98 github.com/urfave/cli.(*App).Run
    18.98 github.com/urfave/cli.(*App).Run
    18.98 . /srchub.lumenfield.work/urfave/cli.(*App).Run
    18.98   /src/cmd/buildkitd/main.go:412
    runtim  /src/cmd/       /src/cmd/buildkitd/main.go:412
    18.98 runtime.goexiruntime.goexit
    18.98 , /usr/local/go/sruntime.goexit
    18.98 time="2025-07-04T15:37:40Z" level=warning mstime="2025-07-04T15:37:40Z" level=warning msg="using host network as the defaultime="2025-07-04T15:37:40Z" level=warning msg="using host network as the default"
    18.98 time="2025-07-04T15:37:40Z" level=info msg="found 1 workers, default=\"jdpy0jn4glj9yrk5rkens8nmm\""
    18.98 `time="2025-07-04T15:37:40Z" level=warning msg="currently, only the default worker can be used."
    18.98 (buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/main.go:368
    18.98 #github.com/urfave/cli.HandleAction
    18.98 . /src/vendor/github.com/urfave/cli/app.go:524
    18.98 !github.com/urfave/cli.(*App).Run
    18.98 . /src/vendor/time="2025-07-04T15:37:40Z" level=info msg="found 1 workers, default=\"jdpy0jn4glj9yrk5rkens8nmm\""
    18.98 `time="2025-07-04T15:37time="2025-07-04T15:37:40Z" level=info msg="found 1 workers, default=\"jdpy0jn4glj9yrk5rkens8nmm\""
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/main.go:368
    18.98 #github.com/urfave/cli.HandleActimain.main.func3
    18.98   /src/cmd/buildkmain.main.func3
    18.98   /src/vendor/github.com/urfave/c /src/vendor/github.com/urfave/cli/      /src/vendor/github.com/urfave/cli/app.go:524
    18.98 main.main
    18.98   /src/cmd/buildmain.main
    18.98   /src/cmd/buildkitd/main.go:main.main
    18.98   /usr/local/go/src/runtime/proc. /usr/local/g    /usr/local/go/src/runtime/proc.go:283
    18.98 time="2025-07-0time="2025-07-04T15:37:47Z" level=info msg="time="2025-07-04T15:37:47Z" level=info msg="auto snapshotter: using overlayfs"
    18.98 time="2025-07-04T15:37:47Z" level=warning msg="skipping containerd worker, as \"/rtime="2025-07-04T15:37:47Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
    18.98 dtime="2025-07-04T15:37:47Z" level=info msg="found 1 workers, default=\"jdpy0jn4glj9yrk5rkens8nmm\""
    18.98 `time="2025-07-04T15:37:47Z" level=warning msg="currently, only the default worker can be used."
    18.98 (buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/main.go:368
    18.98 #github.com/urtime="2025-07-04T15:37:47Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
    18.98 buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/main.go:buildkitd: invalid entitlement : device
    18.98 main.main.func3
    18.98   /src/cmd/buildkitd/mainbuildkitd: invalid entitlement : device
    18.98 github.com/urfavhub.lumenfield.work/urfave/cli.HandleActihub.lumenfield.work/urfave/cli.HandleAction
    18.98   /src/vendor/github.com/urfave/cli/app.go:286
    18.98   /src/vendor/github.com/urfave/cl        /src/vendor/github.com/urfave/cli/app.go:286
    18.98 runtime.maruntime.main
    18.98 ' /usr/localruntime.main
    18.98   /usr/local/go/src/runtime/asm_amd64.s:  /usr/local/go/  /usr/local/go/src/runtime/asm_amd64.s:1700
    ------
    ERROR: Error response from daemon: Container 02c07f72cf1114a36e9974e545863fd2ef208d788110b9437e3433af751ec63a is restarting, wait until the container is running
    
  5. crazy-max commented on Jul 10, 2025

    @crazy-max
    Member

    Thanks for your feedback!

    So I tried method 1 and got this error: ERROR: failed to build: failed to solve: granting entitlement device is not allowed by build daemon configuration.

    Meaning docker build -f Dockerfile --allow device . still error out with granting entitlement device is not allowed by build daemon configuration? I will try to repro on my side.

    Also, if anyone else stumbles across this, the CDI specification created by nvidia-ctk is v0.5.0 so the version field on the YAML spec needs to be manually changed to 0.6.0

    Indeed we should document that as well!

    step due to not picking up the Nvidia drivers: NVIDIA-SMI couldn't find libnvidia-ml.so library in your system. Please make sure that the NVIDIA Display Driver is properly installed and present in your system.

    Ok that would mean that nvidia-ctk doesn't mount the driver correctly. Would mind sharing your YAML CDI spec and also the output of nvidia-ctk --version and nvidia-smi commands from your host please?

    Btw how did you install nvidia-ctk? Did you follow https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#with-apt-ubuntu-debian? In my case I have done:

    apt-get update && apt-get install -y curl gnupg
    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
    curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    apt-get update && apt-get install -y nvidia-container-toolkit
    

    Also, just to confirm, running docker run --rm --runtime=nvidia nvidia/cuda:12.9.1-base-ubuntu22.04 nvidia-smi is able to correctly pick up the GPU.

    In this case it doesn't use CDI but the NVIDIA container runtime.

    I have tried docker buildx create --use --bootstrap --name gpu-builder --buildkitd-flags '--allow-insecure-entitlement device' which resulted in the error below. Appears to not allow for the --allow-insecure-entitlement device' flag.
    ...
    18.98 (buildkitd: invalid entitlement : device

    Yes only security.insecure and network.host are valid insecure entitlements.


    As this is working when using the nvidia container runtime we can try using a container builder that would make the GPU request and create the CDI spec automatically within this container.

    For this we need a custom BuildKit image because current BuildKit release image is based on Alpine that doesn't support NVIDIA drivers. This is tracked in moby/buildkit#5944. For now I have pushed a temp image on Docker Hub: crazymax/buildkit:v0.23.2-ubuntu-nvidia.

    Now you can create your gpubuilder with:

    $ docker buildx create --name nvidia --driver-opt "image=crazymax/buildkit:v0.23.2-ubuntu-nvidia" --buildkitd-flags "--debug" --bootstrap
    #1 [internal] booting buildkit
    #1 pulling image crazymax/buildkit:v0.23.2-ubuntu-nvidia
    #1 pulling image crazymax/buildkit:v0.23.2-ubuntu-nvidia 8.9s done
    #1 creating container buildx_buildkit_gpubuilder0
    #1 creating container buildx_buildkit_gpubuilder0 1.6s done
    #1 DONE 10.5s
    gpubuilder
    

    Inspecting the builder:

    $ docker buildx inspect gpubuilder
    Name:          gpubuilder
    Driver:        docker-container
    Last Activity: 2025-07-10 08:18:09 +0000 UTC
    
    Nodes:
    Name:                  gpubuilder0
    Endpoint:              unix:///var/run/docker.sock
    Driver Options:        image="crazymax/buildkit:v0.23.2-ubuntu-nvidia"
    Status:                running
    BuildKit daemon flags: --allow-insecure-entitlement=network.host
    BuildKit version:      v0.23.2
    Platforms:             linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
    Labels:
     org.mobyproject.buildkit.worker.executor:         oci
     org.mobyproject.buildkit.worker.hostname:         d6aa9cbe8462
     org.mobyproject.buildkit.worker.network:          host
     org.mobyproject.buildkit.worker.oci.process-mode: sandbox
     org.mobyproject.buildkit.worker.selinux.enabled:  false
     org.mobyproject.buildkit.worker.snapshotter:      overlayfs
    Devices:
     Name:      nvidia.com/gpu
     On-Demand: true
    GC Policy rule#0:
     All:            false
     Filters:        type==source.local,type==exec.cachemount,type==source.git.checkout
     Keep Duration:  48h0m0s
     Max Used Space: 488.3MiB
    GC Policy rule#1:
     All:            false
     Keep Duration:  1440h0m0s
     Reserved Space: 9.313GiB
     Max Used Space: 93.13GiB
     Min Free Space: 188.1GiB
    GC Policy rule#2:
     All:            false
     Reserved Space: 9.313GiB
     Max Used Space: 93.13GiB
     Min Free Space: 188.1GiB
    GC Policy rule#3:
     All:            true
     Reserved Space: 9.313GiB
     Max Used Space: 93.13GiB
     Min Free Space: 188.1GiB
    

    We can see nvidia.com/gpu vendor is detected as a device in the builder which means that drivers were detected.

    Optionally you can check if NVIDIA GPU devices are available in the container using nvidia-smi:

    $ docker exec -it buildx_buildkit_gpubuilder0 nvidia-smi -L
    GPU 0: NVIDIA GeForce RTX 3060 Ti (UUID: GPU-63109a83-ae03-3291-b519-c133b6caf78e)
    

    Now you should be able to build your Dockerfile with:

    # syntax=docker/dockerfile:1.17-labs
    FROM ubuntu
    RUN --device=nvidia.com/gpu nvidia-smi -L
    
    $ docker buildx --builder gpubuilder build --progress=plain .
    

    Note that you don't need an nvidia/cuda base image and can just use an ubuntu one as CDI mount the drivers.

    During the build it will generate the CDI spec automatically. You should see smth like this in the build logs:

    #0 building with "gpubuilder" instance using docker-container driver
    ...
    
    #7 preparing device nvidia.com/gpu
    #7 0.000 > apt-get update
    ...
    #7 4.872 > apt-get install -y gpg
    ...
    #7 10.16 Downloading NVIDIA GPG key
    #7 10.21 > apt-get update
    ...
    #7 12.15 > apt-get install -y --no-install-recommends nvidia-container-toolkit-base
    ...
    #7 17.80 time="2025-04-15T08:58:16Z" level=info msg="Generated CDI spec with version 0.8.0"
    #7 DONE 17.8s
    

    You can check the generated CDI specification within the container builder with the following command after the build:

    $ docker exec -it buildx_buildkit_gpubuilder0 cat /etc/cdi/nvidia.yaml
    
  6. oisinturbittcx1 commented on Jul 16, 2025

    @oisinturbittcx1
    Author

    Thank you so much for this! Using the gpubuilder with your image is working perfectly. I have been able to get this working with our workflow and it now allows us to compile CUDA kernels at image build time.

  7. pechersky commented on Jul 18, 2025

    @pechersky

    I tried following the gpubuilder suggestions (note that the comment and thus the commands below actually named the builder nvidia), but am still running into issues, with a Failed to initialize NVML at the end, which then leads to nvidia-smi not found in later build attempts.

    $ docker buildx create --name nvidia --driver-opt "image=crazymax/buildkit:v0.23.2-ubuntu-nvidia" --buildkitd-flags "--debug" --bootstrap
    [+] Building 1.6s (1/1) FINISHED                                                                           
     => [internal] booting buildkit                                                                       1.6s
     => => pulling image crazymax/buildkit:v0.23.2-ubuntu-nvidia                                          0.7s
     => => creating container buildx_buildkit_nvidia0                                                     0.9s
    nvidia
    $ docker buildx inspect nvidia
    Name:          nvidia
    Driver:        docker-container
    Last Activity: 2025-07-18 02:53:14 +0000 UTC
    
    Nodes:
    Name:                  nvidia0
    Endpoint:              unix:///var/run/docker.sock
    Driver Options:        image="crazymax/buildkit:v0.23.2-ubuntu-nvidia"
    Status:                running
    BuildKit daemon flags: --debug --allow-insecure-entitlement=network.host
    BuildKit version:      v0.23.2
    Platforms:             linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
    Labels:
     org.mobyproject.buildkit.worker.executor:         oci
     org.mobyproject.buildkit.worker.hostname:         7e32e54ad9e4
     org.mobyproject.buildkit.worker.network:          host
     org.mobyproject.buildkit.worker.oci.process-mode: sandbox
     org.mobyproject.buildkit.worker.selinux.enabled:  false
     org.mobyproject.buildkit.worker.snapshotter:      overlayfs
    Devices:
     Name:      nvidia.com/gpu
     On-Demand: true
    GC Policy rule#0:
     All:            false
     Filters:        type==source.local,type==exec.cachemount,type==source.git.checkout
     Keep Duration:  48h0m0s
     Max Used Space: 488.3MiB
    GC Policy rule#1:
     All:            false
     Keep Duration:  1440h0m0s
     Reserved Space: 9.313GiB
     Max Used Space: 93.13GiB
     Min Free Space: 188.1GiB
    GC Policy rule#2:
     All:            false
     Reserved Space: 9.313GiB
     Max Used Space: 93.13GiB
     Min Free Space: 188.1GiB
    GC Policy rule#3:
     All:            true
     Reserved Space: 9.313GiB
     Max Used Space: 93.13GiB
     Min Free Space: 188.1GiB
    $ docker exec -it buildx_buildkit_nvidia0 nvidia-smi -L
    GPU 0: NVIDIA RTX A4500 Laptop GPU (UUID: GPU-ade25899-fff9-7c18-98dd-39132689d078)
    $ cat Dockerfile.test 
    # syntax=docker/dockerfile:1.17-labs
    FROM ubuntu
    
    RUN --device=nvidia/gpu=all nvidia-smi -L
    $ docker buildx --builder nvidia build --progress=plain -f Dockerfile.test .    
    #0 building with "nvidia" instance using docker-container driver
    
    #1 [internal] load build definition from Dockerfile.test
    #1 transferring dockerfile: 134B done
    #1 DONE 0.0s
    
    #2 resolve image config for docker-image://docker.io/docker/dockerfile:1.17-labs
    #2 DONE 1.0s
    
    #3 docker-image://docker.io/docker/dockerfile:1.17-labs@sha256:9187104f31e3a002a8a6a3209ea1f937fb7486c093cbbde1e14b0fa0d7e4f1b5
    #3 resolve docker.io/docker/dockerfile:1.17-labs@sha256:9187104f31e3a002a8a6a3209ea1f937fb7486c093cbbde1e14b0fa0d7e4f1b5 0.0s done
    #3 sha256:a8987824096eac360499af15e51f5ecbaa6120cae7d3fa32cfddb6984d449c0d 0B / 14.09MB 0.2s
    #3 sha256:a8987824096eac360499af15e51f5ecbaa6120cae7d3fa32cfddb6984d449c0d 2.10MB / 14.09MB 0.3s
    #3 sha256:a8987824096eac360499af15e51f5ecbaa6120cae7d3fa32cfddb6984d449c0d 4.19MB / 14.09MB 0.5s
    #3 sha256:a8987824096eac360499af15e51f5ecbaa6120cae7d3fa32cfddb6984d449c0d 6.29MB / 14.09MB 0.6s
    #3 sha256:a8987824096eac360499af15e51f5ecbaa6120cae7d3fa32cfddb6984d449c0d 8.39MB / 14.09MB 0.8s
    #3 sha256:a8987824096eac360499af15e51f5ecbaa6120cae7d3fa32cfddb6984d449c0d 9.44MB / 14.09MB 0.9s
    #3 sha256:a8987824096eac360499af15e51f5ecbaa6120cae7d3fa32cfddb6984d449c0d 11.53MB / 14.09MB 1.1s
    #3 sha256:a8987824096eac360499af15e51f5ecbaa6120cae7d3fa32cfddb6984d449c0d 14.09MB / 14.09MB 1.2s done
    #3 extracting sha256:a8987824096eac360499af15e51f5ecbaa6120cae7d3fa32cfddb6984d449c0d 0.1s done
    #3 DONE 1.3s
    
    #4 [internal] load metadata for docker.io/library/ubuntu:latest
    #4 DONE 0.5s
    
    #5 [internal] load .dockerignore
    #5 transferring context: 114B done
    #5 DONE 0.0s
    
    #6 [1/2] FROM docker.io/library/ubuntu:latest@sha256:c4570d2f4665d5d118ae29fb494dee4f8db8fcfaee0e37a2e19b827f399070d3
    #6 resolve docker.io/library/ubuntu:latest@sha256:c4570d2f4665d5d118ae29fb494dee4f8db8fcfaee0e37a2e19b827f399070d3 done
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 0B / 29.72MB 0.2s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 3.15MB / 29.72MB 0.4s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 5.24MB / 29.72MB 0.6s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 7.34MB / 29.72MB 0.7s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 9.44MB / 29.72MB 0.9s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 12.58MB / 29.72MB 1.2s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 15.73MB / 29.72MB 1.4s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 17.83MB / 29.72MB 1.5s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 22.02MB / 29.72MB 1.8s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 24.12MB / 29.72MB 1.9s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 26.21MB / 29.72MB 2.1s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 28.31MB / 29.72MB 2.2s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 29.72MB / 29.72MB 2.3s done
    #6 extracting sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927
    #6 extracting sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 0.5s done
    #6 DONE 2.9s
    
    #7 preparing device nvidia.com/gpu
    #7 0.001 > /usr/bin/nvidia-smi -L
    #7 0.033 GPU 0: NVIDIA RTX A4500 Laptop GPU (UUID: GPU-ade25899-fff9-7c18-98dd-39132689d078)
    #7 0.040 > apt-get update
    #7 0.433 Get:1 http://archive.ubuntu.com/ubuntu noble InRelease [256 kB]
    #7 0.433 Get:2 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
    #7 0.924 Get:3 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [1270 kB]
    #7 0.983 Get:4 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
    #7 1.115 Get:5 http://archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
    #7 1.251 Get:6 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages [1808 kB]
    #7 1.462 Get:7 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [1128 kB]
    #7 1.581 Get:8 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [1820 kB]
    #7 1.657 Get:9 http://archive.ubuntu.com/ubuntu noble/multiverse amd64 Packages [331 kB]
    #7 1.686 Get:10 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages [19.3 MB]
    #7 1.778 Get:11 http://security.ubuntu.com/ubuntu noble-security/multiverse amd64 Packages [23.0 kB]
    #7 3.023 Get:12 http://archive.ubuntu.com/ubuntu noble/restricted amd64 Packages [117 kB]
    #7 3.028 Get:13 http://archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Packages [45.2 kB]
    #7 3.031 Get:14 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1443 kB]
    #7 3.141 Get:15 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1608 kB]
    #7 3.279 Get:16 http://archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [1942 kB]
    #7 3.444 Get:17 http://archive.ubuntu.com/ubuntu noble-backports/universe amd64 Packages [33.0 kB]
    #7 3.446 Get:18 http://archive.ubuntu.com/ubuntu noble-backports/main amd64 Packages [48.8 kB]
    #7 3.718 Fetched 31.6 MB in 4s (8619 kB/s)
    #7 3.718 Reading package lists...
    #7 4.347 > apt-get install -y gpg
    #7 4.351 Reading package lists...
    #7 4.991 Building dependency tree...
    #7 5.142 Reading state information...
    #7 5.288 The following additional packages will be installed:
    #7 5.289   dirmngr gpg-agent gpgconf gpgv libksba8 libreadline8t64 libsqlite3-0
    #7 5.289   pinentry-curses readline-common
    #7 5.290 Suggested packages:
    #7 5.290   gnupg dbus-user-session libpam-systemd pinentry-gnome3 tor keyboxd scdaemon
    #7 5.290   pinentry-doc readline-doc
    #7 5.317 The following NEW packages will be installed:
    #7 5.318   pinentry-curses readline-common
    #7 5.318 bksba8 libreadline8t64 libsqlite3-0
    #7 5.318   pinentry-curses readline-common
    #7 5.318 The following packages will be upgraded:
    #7 5.319   gpgv
    #7 5.629 1 upgraded, 9 newly installed, 0 to remove and 13 not upgraded.
    #7 5.629 Need to get 2444 kB of archives.
    #7 5.629 After this operation, 5597 kB of additional disk space will be used.
    #7 5.629 Get:1 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 gpgv amd64 2.4.4-2ubuntu17.3 [158 kB]
    #7 6.141 Get:2 http://archive.ubuntu.com/ubuntu noble/main amd64 readline-common all 8.2-4build1 [56.5 kB]
    #7 6.166 Get:3 http://archive.ubuntu.com/ubuntu noble/main amd64 libreadline8t64 amd64 8.2-4build1 [153 kB]
    #7 6.278 Get:4 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libsqlite3-0 amd64 3.45.1-1ubuntu2.3 [701 kB]
    #7 6.476 Get:5 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 gpgconf amd64 2.4.4-2ubuntu17.3 [104 kB]
    #7 6.489 Get:6 http://archive.ubuntu.com/ubuntu noble/main amd64 libksba8 amd64 1.6.6-1build1 [122 kB]
    #7 6.552 Get:7 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 dirmngr amd64 2.4.4-2ubuntu17.3 [323 kB]
    #7 6.584 Get:8 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 gpg amd64 2.4.4-2ubuntu17.3 [565 kB]
    #7 6.626 Get:9 http://archive.ubuntu.com/ubuntu noble/main amd64 pinentry-curses amd64 1.2.1-3ubuntu5 [35.2 kB]
    #7 6.628 Get:10 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 gpg-agent amd64 2.4.4-2ubuntu17.3 [227 kB]
    #7 6.764 debconf: delaying package configuration, since apt-utils is not installed
    #7 6.791 Fetched 2444 kB in 1s (1812 kB/s)
    (Reading database ... 8613 files and directories currently installed.)
    #7 6.808 Preparing to unpack .../gpgv_2.4.4-2ubuntu17.3_amd64.deb ...
    #7 6.823 Unpacking gpgv (2.4.4-2ubuntu17.3) over (2.4.4-2ubuntu17.2) ...
    #7 6.867 Setting up gpgv (2.4.4-2ubuntu17.3) ...
    #7 6.906 Selecting previously unselected package readline-common.
    (Reading database ... 8613 files and directories currently installed.)
    #7 6.910 Preparing to unpack .../0-readline-common_8.2-4build1_all.deb ...
    #7 6.914 Unpacking readline-common (8.2-4build1) ...
    #7 6.950 Selecting previously unselected package libreadline8t64:amd64.
    #7 6.951 Preparing to unpack .../1-libreadline8t64_8.2-4build1_amd64.deb ...
    #7 6.962 Adding 'diversion of /lib/x86_64-linux-gnu/libhistory.so.8 to /lib/x86_64-linux-gnu/libhistory.so.8.usr-is-merged by libreadline8t64'
    #7 6.969 Adding 'diversion of /lib/x86_64-linux-gnu/libhistory.so.8.2 to /lib/x86_64-linux-gnu/libhistory.so.8.2.usr-is-merged by libreadline8t64'
    #7 6.974 Adding 'diversion of /lib/x86_64-linux-gnu/libreadline.so.8 to /lib/x86_64-linux-gnu/libreadline.so.8.usr-is-merged by libreadline8t64'
    #7 6.979 Adding 'diversion of /lib/x86_64-linux-gnu/libreadline.so.8.2 to /lib/x86_64-linux-gnu/libreadline.so.8.2.usr-is-merged by libreadline8t64'
    #7 6.983 Unpacking libreadline8t64:amd64 (8.2-4build1) ...
    #7 7.018 Selecting previously unselected package libsqlite3-0:amd64.
    #7 7.019 Preparing to unpack .../2-libsqlite3-0_3.45.1-1ubuntu2.3_amd64.deb ...
    #7 7.024 Unpacking libsqlite3-0:amd64 (3.45.1-1ubuntu2.3) ...
    #7 7.062 Selecting previously unselected package gpgconf.
    #7 7.063 Preparing to unpack .../3-gpgconf_2.4.4-2ubuntu17.3_amd64.deb ...
    #7 7.067 Unpacking gpgconf (2.4.4-2ubuntu17.3) ...
    #7 7.107 Selecting previously unselected package libksba8:amd64.
    #7 7.108 Preparing to unpack .../4-libksba8_1.6.6-1build1_amd64.deb ...
    #7 7.112 Unpacking libksba8:amd64 (1.6.6-1build1) ...
    #7 7.154 Selecting previously unselected package dirmngr.
    #7 7.154 Preparing to unpack .../5-dirmngr_2.4.4-2ubuntu17.3_amd64.deb ...
    #7 7.169 Unpacking dirmngr (2.4.4-2ubuntu17.3) ...
    #7 7.199 Selecting previously unselected package gpg.
    #7 7.199 Preparing to unpack .../6-gpg_2.4.4-2ubuntu17.3_amd64.deb ...
    #7 7.203 Unpacking gpg (2.4.4-2ubuntu17.3) ...
    #7 7.239 Selecting previously unselected package pinentry-curses.
    #7 7.240 Preparing to unpack .../7-pinentry-curses_1.2.1-3ubuntu5_amd64.deb ...
    #7 7.244 Unpacking pinentry-curses (1.2.1-3ubuntu5) ...
    #7 7.280 Selecting previously unselected package gpg-agent.
    #7 7.281 Preparing to unpack .../8-gpg-agent_2.4.4-2ubuntu17.3_amd64.deb ...
    #7 7.285 Unpacking gpg-agent (2.4.4-2ubuntu17.3) ...
    #7 7.320 Setting up libksba8:amd64 (1.6.6-1build1) ...
    #7 7.331 Setting up pinentry-curses (1.2.1-3ubuntu5) ...
    #7 7.350 Setting up libsqlite3-0:amd64 (3.45.1-1ubuntu2.3) ...
    #7 7.362 Setting up readline-common (8.2-4build1) ...
    #7 7.376 Setting up libreadline8t64:amd64 (8.2-4build1) ...
    #7 7.389 Setting up gpgconf (2.4.4-2ubuntu17.3) ...
    #7 7.401 Setting up gpg (2.4.4-2ubuntu17.3) ...
    #7 7.413 Setting up gpg-agent (2.4.4-2ubuntu17.3) ...
    #7 7.747 Setting up dirmngr (2.4.4-2ubuntu17.3) ...
    #7 7.858 Processing triggers for libc-bin (2.39-0ubuntu8.4) ...
    #7 7.904 Downloading NVIDIA GPG key
    #7 8.183 > apt-get update
    #7 8.461 Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64  InRelease [1581 B]
    #7 8.466 Hit:2 http://archive.ubuntu.com/ubuntu noble InRelease
    #7 8.529 Hit:3 http://security.ubuntu.com/ubuntu noble-security InRelease
    #7 8.534 Get:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64  Packages [737 kB]
    #7 8.562 Hit:5 http://archive.ubuntu.com/ubuntu noble-updates InRelease
    #7 8.658 Hit:6 http://archive.ubuntu.com/ubuntu noble-backports InRelease
    #7 8.703 Fetched 739 kB in 1s (1460 kB/s)
    #7 8.703 Reading package lists...
    #7 9.319 > apt-get install -y --no-install-recommends nvidia-container-toolkit-base
    #7 9.322 Reading package lists...
    #7 9.925 Building dependency tree...
    #7 10.10 Reading state information...
    #7 10.26   nvidia-container-toolkit-base
    #7 10.26 be installed:
    #7 10.26   nvidia-container-toolkit-base
    #7 10.54 0 upgraded, 1 newly installed, 0 to remove and 13 not upgraded.
    #7 10.54 Need to get 3797 kB of archives.
    #7 10.54 After this operation, 20.7 MB of additional disk space will be used.
    #7 10.54 Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64  nvidia-container-toolkit-base 1.17.8-1 [3797 kB]
    #7 10.99 debconf: delaying package configuration, since apt-utils is not installed
    #7 11.01 Fetched 3797 kB in 1s (5812 kB/s)
    #7 11.03 Selecting previously unselected package nvidia-container-toolkit-base.
    (Reading database ... 8732 files and directories currently installed.)
    #7 11.03 Preparing to unpack .../nvidia-container-toolkit-base_1.17.8-1_amd64.deb ...
    #7 11.04 Unpacking nvidia-container-toolkit-base (1.17.8-1) ...
    #7 11.30 Setting up nvidia-container-toolkit-base (1.17.8-1) ...
    #7 11.36 time="2025-07-18T02:55:21Z" level=info msg="Using /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.1"
    #7 11.39 time="2025-07-18T02:55:21Z" level=info msg="Auto-detected mode as 'wsl'"
    #7 11.39 time="2025-07-18T02:55:21Z" level=info msg="Selecting /dev/dxg as /dev/dxg"
    #7 11.39 time="2025-07-18T02:55:21Z" level=info msg="Using WSL driver store paths: [/usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68 /usr/lib/wsl/drivers/iigd_dch.inf_amd64_b535843ef4c7897f]"
    #7 11.39 time="2025-07-18T02:55:21Z" level=warning msg="Found multiple driver store paths: [/usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68 /usr/lib/wsl/drivers/iigd_dch.inf_amd64_b535843ef4c7897f]"
    #7 11.39 time="2025-07-18T02:55:21Z" level=info msg="Selecting /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libcuda.so.1.1 as /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libcuda.so.1.1"
    #7 11.39 time="2025-07-18T02:55:21Z" level=info msg="Selecting /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libcuda_loader.so as /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libcuda_loader.so"
    #7 11.40 time="2025-07-18T02:55:21Z" level=info msg="Selecting /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvidia-ptxjitcompiler.so.1 as /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvidia-ptxjitcompiler.so.1"
    #7 11.40 time="2025-07-18T02:55:21Z" level=info msg="Selecting /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvidia-ml.so.1 as /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvidia-ml.so.1"
    #7 11.40 time="2025-07-18T02:55:21Z" level=warning msg="Could not locate libdxcore.so: pattern libdxcore.so not found"
    #7 11.40 nvidia-ml_loader.so as /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvidia-ml_loader.so"
    #7 11.40 time="2025-07-18T02:55:21Z" level=warning msg="Could not locate libdxcore.so: pattern libdxcore.so not found"
    #7 11.40 time="2025-07-18T02:55:21Z" level=info msg="Selecting /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvdxgdmal.so.1 as /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvdxgdmal.so.1"
    #7 11.40 time="2025-07-18T02:55:21Z" level=info msg="Selecting /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/nvcubins.bin as /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/nvcubins.bin"
    #7 11.40 time="2025-07-18T02:55:21Z" level=info msg="Selecting /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/nvidia-smi as /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/nvidia-smi"
    #7 11.40 time="2025-07-18T02:55:21Z" level=info msg="Generated CDI spec with version 0.8.0"
    #7 DONE 11.4s
    
    #8 [2/2] RUN --device=nvidia/gpu=all nvidia-smi -L
    #8 0.157 Failed to initialize NVML: N/A
    #8 ERROR: process "/bin/sh -c nvidia-smi -L" did not complete successfully: exit code: 3
    ------
     > [2/2] RUN --device=nvidia/gpu=all nvidia-smi -L:
    0.157 Failed to initialize NVML: N/A
    ------
    WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
    Dockerfile.test:4
    --------------------
       2 |     FROM ubuntu
       3 |     
       4 | >>> RUN --device=nvidia/gpu=all nvidia-smi -L
       5 |     
    --------------------
    ERROR: failed to solve: process "/bin/sh -c nvidia-smi -L" did not complete successfully: exit code: 3
    $ docker exec -it buildx_buildkit_nvidia0 cat /etc/cdi/nvidia.yaml    
    ---
    cdiVersion: 0.3.0
    containerEdits:
      env:
      - NVIDIA_VISIBLE_DEVICES=void
      hooks:
      - args:
        - nvidia-cdi-hook
        - create-symlinks
        - --link
        - /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/nvidia-smi::/usr/bin/nvidia-smi
        env:
        - NVIDIA_CTK_DEBUG=false
        hookName: createContainer
        path: /usr/bin/nvidia-cdi-hook
      - args:
        - nvidia-cdi-hook
        - update-ldcache
        - --folder
        - /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68
        env:
        - NVIDIA_CTK_DEBUG=false
        hookName: createContainer
        path: /usr/bin/nvidia-cdi-hook
      mounts:
      - containerPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libcuda.so.1.1
        hostPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libcuda.so.1.1
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libcuda_loader.so
        hostPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libcuda_loader.so
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvdxgdmal.so.1
        hostPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvdxgdmal.so.1
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvidia-ml.so.1
        hostPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvidia-ml.so.1
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvidia-ml_loader.so
        hostPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvidia-ml_loader.so
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvidia-ptxjitcompiler.so.1
        hostPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/libnvidia-ptxjitcompiler.so.1
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/nvcubins.bin
        hostPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/nvcubins.bin
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/nvidia-smi
        hostPath: /usr/lib/wsl/drivers/nvlt.inf_amd64_3b4f6ef9d0e51d68/nvidia-smi
        options:
        - ro
        - nosuid
        - nodev
        - bind
    devices:
    - containerEdits:
        deviceNodes:
        - path: /dev/dxg
      name: all
    kind: nvidia.com/gpu
    $ # Trying build again
    $ docker buildx --builder nvidia build --progress=plain -f Dockerfile.test .
    #0 building with "nvidia" instance using docker-container driver
    
    #1 [internal] load build definition from Dockerfile.test
    #1 DONE 0.0s
    
    #1 [internal] load build definition from Dockerfile.test
    #1 transferring dockerfile: 134B done
    #1 DONE 0.0s
    
    #2 resolve image config for docker-image://docker.io/docker/dockerfile:1.17-labs
    #2 DONE 0.3s
    
    #3 docker-image://docker.io/docker/dockerfile:1.17-labs@sha256:9187104f31e3a002a8a6a3209ea1f937fb7486c093cbbde1e14b0fa0d7e4f1b5
    #3 resolve docker.io/docker/dockerfile:1.17-labs@sha256:9187104f31e3a002a8a6a3209ea1f937fb7486c093cbbde1e14b0fa0d7e4f1b5 0.0s done
    #3 CACHED
    
    #4 [internal] load metadata for docker.io/library/ubuntu:latest
    #4 DONE 0.1s
    
    #5 [internal] load .dockerignore
    #5 transferring context: 114B done
    #5 DONE 0.0s
    
    #6 [1/2] FROM docker.io/library/ubuntu:latest@sha256:c4570d2f4665d5d118ae29fb494dee4f8db8fcfaee0e37a2e19b827f399070d3
    #6 resolve docker.io/library/ubuntu:latest@sha256:c4570d2f4665d5d118ae29fb494dee4f8db8fcfaee0e37a2e19b827f399070d3 0.0s done
    #6 CACHED
    
    #7 [2/2] RUN --device=nvidia/gpu=all nvidia-smi -L
    #7 0.043 /bin/sh: 1: nvidia-smi: not found
    #7 ERROR: process "/bin/sh -c nvidia-smi -L" did not complete successfully: exit code: 127
    ------
     > [2/2] RUN --device=nvidia/gpu=all nvidia-smi -L:
    0.043 /bin/sh: 1: nvidia-smi: not found
    ------
    WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
    Dockerfile.test:4
    --------------------
       2 |     FROM ubuntu
       3 |     
       4 | >>> RUN --device=nvidia/gpu=all nvidia-smi -L
       5 |     
    --------------------
    ERROR: failed to solve: process "/bin/sh -c nvidia-smi -L" did not complete successfully: exit code: 127
    
  8. crazy-max commented on Jul 18, 2025

    @crazy-max
    Member

    Thank you so much for this! Using the gpubuilder with your image is working perfectly. I have been able to get this working with our workflow and it now allows us to compile CUDA kernels at image build time.

    @oisinturbittcx1 Awesome! Would you mind sharing your YAML CDI spec and also the output of nvidia-ctk --version and nvidia-smi commands from your host please? So we can check this issue?

    I tried following the gpubuilder suggestions (note that the comment and thus the commands below actually named the builder nvidia), but am still running into issues, with a Failed to initialize NVML at the end, which then leads to nvidia-smi not found in later build attempts.

    @pechersky We had the same issues before on WSL but we solved this by making a device request for the container builder #3063. But since recent version on WSL it doesn't work anymore 😢.

    It seems to be a limitation mounting dxcore within the container. Here we are installing the cuda drivers for /dev/dxg device but it doesn't seem enough to make it works as libdxcore.so is missing as you can see in the logs:

    #7 11.40 time="2025-07-18T02:55:21Z" level=warning msg="Could not locate libdxcore.so: pattern libdxcore.so not found"
    

    If we look at /usr/lib/wsl/lib content we have on the host:

    drwxr-xr-x 1 root root        60 Feb  9 11:48 .
    drwxr-xr-x 4 root root      4096 Aug  1  2021 ..
    -r-xr-xr-x 4 root root    162664 Dec  4 02:11 libcuda.so
    -r-xr-xr-x 4 root root    162664 Dec  4 02:11 libcuda.so.1
    -r-xr-xr-x 4 root root    162664 Dec  4 02:11 libcuda.so.1.1
    -r-xr-xr-x 1 root root  10362952 Nov  7 03:41 libcudadebugger.so.1
    -r-xr-xr-x 1 root root    801840 Oct 20  2023 libd3d12.so
    -r-xr-xr-x 1 root root   6880344 Oct 20  2023 libd3d12core.so
    -r-xr-xr-x 1 root root    942048 Mar 25  2024 libdxcore.so
    -r-xr-xr-x 3 root root  15601040 Dec  4 02:11 libnvcuvid.so
    -r-xr-xr-x 3 root root  15601040 Dec  4 02:11 libnvcuvid.so.1
    -r-xr-xr-x 1 root root 133120256 Nov  7 03:41 libnvdxdlkernels.so
    -r-xr-xr-x 3 root root    608872 Dec  4 02:11 libnvidia-encode.so
    -r-xr-xr-x 3 root root    608872 Dec  4 02:11 libnvidia-encode.so.1
    -r-xr-xr-x 1 root root    249904 Nov  7 03:41 libnvidia-ml.so.1
    -r-xr-xr-x 3 root root    387536 Dec  4 02:11 libnvidia-opticalflow.so
    -r-xr-xr-x 3 root root    387536 Dec  4 02:11 libnvidia-opticalflow.so.1
    -r-xr-xr-x 1 root root     72664 Nov  7 03:41 libnvoptix.so.1
    lrwxrwxrwx 1 root root        15 Feb  9 11:48 libnvoptix_loader.so.1 -> libnvoptix.so.1
    -r-xr-xr-x 2 root root  81114696 Dec  4 02:11 libnvwgf2umx.so
    -r-xr-xr-x 2 root root    738304 Dec  4 02:11 nvidia-smi
    

    I tried to find the package that installs these files in the WSL Ubuntu distribution (dpkg -S) but seems there are none.

    Looking at the docs https://docs.nvidia.com/cuda//wsl-user-guide/index.html?utm_source=chatgpt.com#cuda-support-for-wsl-2, these files are available across all WSL distributions when drivers are installed on the host (Windows):

    Once a Windows NVIDIA GPU driver is installed on the system, CUDA becomes available within WSL 2. The CUDA driver installed on Windows host will be stubbed inside the WSL 2 as libcuda.so, therefore users must not install any NVIDIA GPU Linux driver within WSL 2. One has to be very careful here as the default CUDA Toolkit comes packaged with a driver, and it is easy to overwrite the WSL 2 NVIDIA driver with the default installation. We recommend developers to use a separate CUDA Toolkit for WSL 2 (Ubuntu) available from the CUDA Toolkit Downloads page to avoid this overwriting. This WSL-Ubuntu CUDA toolkit installer will not overwrite the NVIDIA driver that was already mapped into the WSL 2 environment. To learn how to compile CUDA applications, please read the CUDA documentation for Linux.

    There are the meta packages as stated in "Option 2" but no linux drivers.

    Mounting /usr/lib/wsl in BuildKit container seems to work https://github.com/docker/buildx/compare/master...crazy-max:buildx:driver-ctn-mount-wsl-lib?expand=1:

    $ docker buildx create --name gpubuilder --driver-opt "image=crazymax/buildkit:v0.23.2-ubuntu-nvidia" --buildkitd-flags "--debug" --bootstrap
    #1 [internal] booting buildkit
    #1 pulling image crazymax/buildkit:v0.23.2-ubuntu-nvidia
    #1 pulling image crazymax/buildkit:v0.23.2-ubuntu-nvidia 8.9s done
    #1 creating container buildx_buildkit_gpubuilder0
    #1 creating container buildx_buildkit_gpubuilder0 1.6s done
    #1 DONE 10.5s
    gpubuilder
    
    $ docker inspect buildx_buildkit_gpubuilder0
    ...
            "Mounts": [
                {
                    "Type": "volume",
                    "Name": "buildx_buildkit_gpubuilder0_state",
                    "Source": "/var/lib/docker/volumes/buildx_buildkit_gpubuilder0_state/_data",
                    "Destination": "/var/lib/buildkit",
                    "Driver": "local",
                    "Mode": "z",
                    "RW": true,
                    "Propagation": ""
                },
                {
                    "Type": "bind",
                    "Source": "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/f969f076ad437ac0162d56bf356c2a910d8ddd313a6ac1d90246b6b64d06d59a",
                    "Destination": "/usr/lib/wsl",
                    "Mode": "",
                    "RW": false,
                    "Propagation": "rprivate"
                }
            ],
    ...
    
    $ docker exec -it buildx_buildkit_gpubuilder0 nvidia-smi
    Fri Feb 14 14:43:38 2025
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 565.77.01              Driver Version: 566.36         CUDA Version: 12.7     |
    |-----------------------------------------+------------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
    |                                         |                        |               MIG M. |
    |=========================================+========================+======================|
    |   0  NVIDIA GeForce RTX 3060 Ti     On  |   00000000:2A:00.0  On |                  N/A |
    |  0%   52C    P8             22W /  200W |    3696MiB /   8192MiB |      8%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
    
    +-----------------------------------------------------------------------------------------+
    | Processes:                                                                              |
    |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
    |        ID   ID                                                               Usage      |
    |=========================================================================================|
    |    0   N/A  N/A        25      G   /Xwayland                                   N/A      |
    +-----------------------------------------------------------------------------------------+
    

    Maybe this could be a workaround for WSL case but mounting from the host could be error prone if a remote context/builder is used. Could maybe be an opt-in or we could detect if we are using the default context and allow it in such case.

  9. oisinturbittcx1 commented on Jul 18, 2025

    @oisinturbittcx1
    Author

    @crazy-max I got the above working on a machine that I no longer have access to unfortunately. But to do it, I followed the instructions here to install nvidia-ckt and get the following running nvidia-ckt --version:

    nvidia-ctk --version
    NVIDIA Container Toolkit CLI version 1.17.8
    commit: f202b80a9b9d0db00d9b1d73c0128c8962c55f4d

    Which creates the /etc/cdi/nvidia.yaml contents:

    ---
    cdiVersion: 0.5.0
    containerEdits:
      deviceNodes:
      - path: /dev/nvidia-modeset
      - path: /dev/nvidia-uvm
      - path: /dev/nvidia-uvm-tools
      - path: /dev/nvidiactl
      env:
      - NVIDIA_VISIBLE_DEVICES=void
      hooks:
      - args:
        - nvidia-cdi-hook
        - create-symlinks
        - --link
        - libcuda.so.1::/usr/lib/x86_64-linux-gnu/libcuda.so
        env:
        - NVIDIA_CTK_DEBUG=false
        hookName: createContainer
        path: /usr/bin/nvidia-cdi-hook
      - args:
        - nvidia-cdi-hook
        - enable-cuda-compat
        - --host-driver-version=575.57.08
        env:
        - NVIDIA_CTK_DEBUG=false
        hookName: createContainer
        path: /usr/bin/nvidia-cdi-hook
      - args:
        - nvidia-cdi-hook
        - update-ldcache
        - --folder
        - /usr/lib/x86_64-linux-gnu
        env:
        - NVIDIA_CTK_DEBUG=false
        hookName: createContainer
        path: /usr/bin/nvidia-cdi-hook
      mounts:
      - containerPath: /run/nvidia-persistenced/socket
        hostPath: /run/nvidia-persistenced/socket
        options:
        - ro
        - nosuid
        - nodev
        - bind
        - noexec
      - containerPath: /usr/bin/nvidia-cuda-mps-control
        hostPath: /usr/bin/nvidia-cuda-mps-control
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/bin/nvidia-cuda-mps-server
        hostPath: /usr/bin/nvidia-cuda-mps-server
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/bin/nvidia-debugdump
        hostPath: /usr/bin/nvidia-debugdump
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/bin/nvidia-persistenced
        hostPath: /usr/bin/nvidia-persistenced
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/bin/nvidia-smi
        hostPath: /usr/bin/nvidia-smi
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08
        hostPath: /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libcudadebugger.so.575.57.08
        hostPath: /usr/lib/x86_64-linux-gnu/libcudadebugger.so.575.57.08
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.575.57.08
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.575.57.08
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.575.57.08
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.575.57.08
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.575.57.08
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.575.57.08
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.575.57.08
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.575.57.08
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.575.57.08
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.575.57.08
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.575.57.08
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.575.57.08
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.575.57.08
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.575.57.08
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /lib/firmware/nvidia/575.57.08/gsp_ga10x.bin
        hostPath: /lib/firmware/nvidia/575.57.08/gsp_ga10x.bin
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /lib/firmware/nvidia/575.57.08/gsp_tu10x.bin
        hostPath: /lib/firmware/nvidia/575.57.08/gsp_tu10x.bin
        options:
        - ro
        - nosuid
        - nodev
        - bind
    devices:
    - containerEdits:
        deviceNodes:
        - path: /dev/nvidia0
        - path: /dev/dri/card1
        - path: /dev/dri/renderD128
        hooks:
        - args:
          - nvidia-cdi-hook
          - create-symlinks
          - --link
          - ../card1::/dev/dri/by-path/pci-0000:00:1e.0-card
          - --link
          - ../renderD128::/dev/dri/by-path/pci-0000:00:1e.0-render
          env:
          - NVIDIA_CTK_DEBUG=false
          hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
        - args:
          - nvidia-cdi-hook
          - chmod
          - --mode
          - "755"
          - --path
          - /dev/dri
          env:
          - NVIDIA_CTK_DEBUG=false
          hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
      name: "0"
    - containerEdits:
        deviceNodes:
        - path: /dev/nvidia0
        - path: /dev/dri/card1
        - path: /dev/dri/renderD128
        hooks:
        - args:
          - nvidia-cdi-hook
          - create-symlinks
          - --link
          - ../card1::/dev/dri/by-path/pci-0000:00:1e.0-card
          - --link
          - ../renderD128::/dev/dri/by-path/pci-0000:00:1e.0-render
          env:
          - NVIDIA_CTK_DEBUG=false
          hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
        - args:
          - nvidia-cdi-hook
          - chmod
          - --mode
          - "755"
          - --path
          - /dev/dri
          env:
          - NVIDIA_CTK_DEBUG=false
          hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
      name: GPU-e603dd39-6e69-7ba5-7bd0-86d4d3026480
    - containerEdits:
        deviceNodes:
        - path: /dev/nvidia0
        - path: /dev/dri/card1
        - path: /dev/dri/renderD128
        hooks:
        - args:
          - nvidia-cdi-hook
          - create-symlinks
          - --link
          - ../card1::/dev/dri/by-path/pci-0000:00:1e.0-card
          - --link
          - ../renderD128::/dev/dri/by-path/pci-0000:00:1e.0-render
          env:
          - NVIDIA_CTK_DEBUG=false
          hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
        - args:
          - nvidia-cdi-hook
          - chmod
          - --mode
          - "755"
          - --path
          - /dev/dri
          env:
          - NVIDIA_CTK_DEBUG=false
          hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
      name: all
    kind: nvidia.com/gpu
    

    Here is the log of the CDI spec being generated:

    INFO[0000] Using /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.575.57.08 
    INFO[0001] Auto-detected mode as 'nvml'                 
    INFO[0003] Selecting /dev/nvidia0 as /dev/nvidia0       
    INFO[0003] Selecting /dev/dri/card1 as /dev/dri/card1   
    INFO[0003] Selecting /dev/dri/renderD128 as /dev/dri/renderD128 
    INFO[0005] Using driver version 575.57.08               
    INFO[0005] Selecting /dev/nvidia-modeset as /dev/nvidia-modeset 
    INFO[0005] Selecting /dev/nvidia-uvm-tools as /dev/nvidia-uvm-tools 
    INFO[0005] Selecting /dev/nvidia-uvm as /dev/nvidia-uvm 
    INFO[0005] Selecting /dev/nvidiactl as /dev/nvidiactl   
    WARN[0005] Could not locate libnvidia-egl-gbm.so.*.*: pattern libnvidia-egl-gbm.so.*.* not found
    libnvidia-egl-gbm.so.*.*: not found 
    WARN[0005] Could not locate libnvidia-egl-wayland.so.*.*: pattern libnvidia-egl-wayland.so.*.* not found
    libnvidia-egl-wayland.so.*.*: not found 
    WARN[0005] Could not locate libnvidia-allocator.so.575.57.08: pattern libnvidia-allocator.so.575.57.08 not found
    libnvidia-allocator.so.575.57.08: not found 
    WARN[0005] Could not locate libnvidia-vulkan-producer.so.575.57.08: pattern libnvidia-vulkan-producer.so.575.57.08 not found
    libnvidia-vulkan-producer.so.575.57.08: not found 
    WARN[0005] Could not locate nvidia_drv.so: pattern nvidia_drv.so not found 
    WARN[0005] Could not locate libglxserver_nvidia.so.575.57.08: pattern libglxserver_nvidia.so.575.57.08 not found 
    WARN[0005] Could not locate glvnd/egl_vendor.d/10_nvidia.json: pattern glvnd/egl_vendor.d/10_nvidia.json not found 
    WARN[0005] Could not locate egl/egl_external_platform.d/15_nvidia_gbm.json: pattern egl/egl_external_platform.d/15_nvidia_gbm.json not found 
    WARN[0005] Could not locate egl/egl_external_platform.d/10_nvidia_wayland.json: pattern egl/egl_external_platform.d/10_nvidia_wayland.json not found 
    WARN[0005] Could not locate nvidia/nvoptix.bin: pattern nvidia/nvoptix.bin not found 
    WARN[0005] Could not locate X11/xorg.conf.d/10-nvidia.conf: pattern X11/xorg.conf.d/10-nvidia.conf not found 
    WARN[0005] Could not locate X11/xorg.conf.d/nvidia-drm-outputclass.conf: pattern X11/xorg.conf.d/nvidia-drm-outputclass.conf not found 
    WARN[0005] Could not locate vulkan/icd.d/nvidia_icd.json: pattern vulkan/icd.d/nvidia_icd.json not found
    pattern vulkan/icd.d/nvidia_icd.json not found 
    WARN[0005] Could not locate vulkan/icd.d/nvidia_layers.json: pattern vulkan/icd.d/nvidia_layers.json not found
    pattern vulkan/icd.d/nvidia_layers.json not found 
    WARN[0005] Could not locate vulkan/implicit_layer.d/nvidia_layers.json: pattern vulkan/implicit_layer.d/nvidia_layers.json not found
    pattern vulkan/implicit_layer.d/nvidia_layers.json not found 
    INFO[0005] Selecting /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08 as /usr/lib/x86_64-linux-gnu/libcuda.so.575.57.08 
    INFO[0005] Selecting /usr/lib/x86_64-linux-gnu/libcudadebugger.so.575.57.08 as /usr/lib/x86_64-linux-gnu/libcudadebugger.so.575.57.08 
    INFO[0005] Selecting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.575.57.08 as /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.575.57.08 
    INFO[0005] Selecting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.575.57.08 as /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.575.57.08 
    INFO[0005] Selecting /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.575.57.08 as /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.575.57.08 
    INFO[0005] Selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.575.57.08 as /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.575.57.08 
    INFO[0005] Selecting /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.575.57.08 as /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.575.57.08 
    INFO[0005] Selecting /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.575.57.08 as /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.575.57.08 
    INFO[0005] Selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.575.57.08 as /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.575.57.08 
    INFO[0005] Selecting /run/nvidia-persistenced/socket as /run/nvidia-persistenced/socket 
    WARN[0005] Could not locate /nvidia-fabricmanager/socket: pattern /nvidia-fabricmanager/socket not found 
    WARN[0005] Could not locate /tmp/nvidia-mps: pattern /tmp/nvidia-mps not found 
    INFO[0005] Selecting /lib/firmware/nvidia/575.57.08/gsp_ga10x.bin as /lib/firmware/nvidia/575.57.08/gsp_ga10x.bin 
    INFO[0005] Selecting /lib/firmware/nvidia/575.57.08/gsp_tu10x.bin as /lib/firmware/nvidia/575.57.08/gsp_tu10x.bin 
    INFO[0005] Selecting /usr/bin/nvidia-smi as /usr/bin/nvidia-smi 
    INFO[0005] Selecting /usr/bin/nvidia-debugdump as /usr/bin/nvidia-debugdump 
    INFO[0005] Selecting /usr/bin/nvidia-persistenced as /usr/bin/nvidia-persistenced 
    INFO[0005] Selecting /usr/bin/nvidia-cuda-mps-control as /usr/bin/nvidia-cuda-mps-control 
    INFO[0005] Selecting /usr/bin/nvidia-cuda-mps-server as /usr/bin/nvidia-cuda-mps-server 
    WARN[0005] Could not locate nvidia-imex: pattern nvidia-imex not found 
    WARN[0005] Could not locate nvidia-imex-ctl: pattern nvidia-imex-ctl not found 
    WARN[0005] Could not locate libnvidia-egl-gbm.so.*.*: pattern libnvidia-egl-gbm.so.*.* not found
    libnvidia-egl-gbm.so.*.*: not found 
    WARN[0005] Could not locate libnvidia-egl-wayland.so.*.*: pattern libnvidia-egl-wayland.so.*.* not found
    libnvidia-egl-wayland.so.*.*: not found 
    WARN[0005] Could not locate libnvidia-allocator.so.575.57.08: pattern libnvidia-allocator.so.575.57.08 not found
    libnvidia-allocator.so.575.57.08: not found 
    WARN[0005] Could not locate libnvidia-vulkan-producer.so.575.57.08: pattern libnvidia-vulkan-producer.so.575.57.08 not found
    libnvidia-vulkan-producer.so.575.57.08: not found 
    WARN[0005] Could not locate nvidia_drv.so: pattern nvidia_drv.so not found 
    WARN[0005] Could not locate libglxserver_nvidia.so.575.57.08: pattern libglxserver_nvidia.so.575.57.08 not found 
    INFO[0005] Generated CDI spec with version 0.8.0 
    

    There appears to be a mismatch between the log line INFO[0005] Generated CDI spec with version 0.8.0 and the cdiVersion: 0.5.0 in the actual YAML file

  10. oisinturbittcx1 commented on Jul 18, 2025

    @oisinturbittcx1
    Author

    @crazy-max Working your instructions through to the end on this new machine results in a new error (see below). Appears to be that the Nvidia driver versions in crazymax/buildkit:v0.23.2-ubuntu-nvidia is the desktop variant 575.64.03 while this new machine has the server variants 575.57.08:

    #0 building with "gpu-builder" instance using docker-container driver
    
    #1 [internal] load build definition from Dockerfile.test
    #1 transferring dockerfile: 133B done
    #1 DONE 0.0s
    
    #2 resolve image config for docker-image://docker.io/docker/dockerfile:1.17-labs
    #2 DONE 0.1s
    
    #3 docker-image://docker.io/docker/dockerfile:1.17-labs@sha256:9187104f31e3a002a8a6a3209ea1f937fb7486c093cbbde1e14b0fa0d7e4f1b5
    #3 resolve docker.io/docker/dockerfile:1.17-labs@sha256:9187104f31e3a002a8a6a3209ea1f937fb7486c093cbbde1e14b0fa0d7e4f1b5 0.0s done
    #3 CACHED
    
    #4 [internal] load metadata for docker.io/library/ubuntu:latest
    #4 DONE 0.2s
    
    #5 [internal] load .dockerignore
    #5 transferring context: 189B done
    #5 DONE 0.0s
    
    #6 [1/2] FROM docker.io/library/ubuntu:latest@sha256:c4570d2f4665d5d118ae29fb494dee4f8db8fcfaee0e37a2e19b827f399070d3
    #6 resolve docker.io/library/ubuntu:latest@sha256:c4570d2f4665d5d118ae29fb494dee4f8db8fcfaee0e37a2e19b827f399070d3 0.0s done
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 4.19MB / 29.72MB 0.2s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 13.63MB / 29.72MB 0.3s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 24.12MB / 29.72MB 0.5s
    #6 sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 29.72MB / 29.72MB 0.6s done
    #6 extracting sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927
    #6 extracting sha256:32f112e3802cadcab3543160f4d2aa607b3cc1c62140d57b4f5441384f40e927 0.7s done
    #6 DONE 1.4s
    
    #7 preparing device nvidia.com/gpu
    #7 0.001 > apt-get update
    #7 0.145 Hit:1 http://archive.ubuntu.com/ubuntu noble InRelease
    #7 0.145 Get:2 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
    #7 0.150 Get:3 http://archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
    #7 0.176 Get:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64  InRelease [1581 B]
    #7 0.277 Get:5 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
    #7 0.307 Get:6 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1443 kB]
    #7 0.326 Get:7 http://archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [1942 kB]
    #7 0.338 Get:8 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1608 kB]
    #7 0.427 Get:9 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64  Packages [737 kB]
    #7 0.657 Get:10 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [1270 kB]
    #7 0.999 Get:11 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [1128 kB]
    #7 1.043 Get:12 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [1820 kB]
    #7 1.212 Fetched 10.3 MB in 1s (9526 kB/s)
    #7 1.212 Reading package lists...
    #7 2.177 > apt-get install -y gpg
    #7 2.185 Reading package lists...
    #7 3.023 Building dependency tree...
    #7 3.217 Reading state information...
    #7 3.461 gpg is already the newest version (2.4.4-2ubuntu17.3).
    #7 3.461 0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.
    #7 3.463 > apt-get update
    #7 3.508 Hit:1 http://security.ubuntu.com/ubuntu noble-security InRelease
    #7 3.526 Hit:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64  InRelease
    #7 3.643 Hit:3 http://archive.ubuntu.com/ubuntu noble InRelease
    #7 3.715 Hit:4 http://archive.ubuntu.com/ubuntu noble-updates InRelease
    #7 3.787 Hit:5 http://archive.ubuntu.com/ubuntu noble-backports InRelease
    #7 3.850 Reading package lists...
    #7 4.692 > apt-get install -y --no-install-recommends nvidia-container-toolkit-base
    #7 4.698 Reading package lists...
    #7 5.535 Building dependency tree...
    #7 5.725 Reading state information...
    #7 5.952 nvidia-container-toolkit-base is already the newest version (1.17.8-1).
    #7 5.952 0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.
    #7 6.009 time="2025-07-18T09:42:39Z" level=info msg="Using /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.575.64.03"
    #7 6.055 time="2025-07-18T09:42:39Z" level=info msg="Auto-detected mode as 'nvml'"
    #7 6.057 time="2025-07-18T09:42:39Z" level=error msg="failed to generate CDI spec: failed to create device CDI specs: failed to initialize NVML: Driver/library version mismatch"
    #7 ERROR: failed to generate CDI spec: exit status 1
    
    #8 [2/2] RUN --device=nvidia.com/gpu nvidia-smi -L
    #8 ERROR: failed to create nvidia.com/gpu device: failed to generate CDI spec: exit status 1
    ------
     > preparing device nvidia.com/gpu:
    3.715 Hit:4 http://archive.ubuntu.com/ubuntu noble-updates InRelease
    3.787 Hit:5 http://archive.ubuntu.com/ubuntu noble-backports InRelease
    
    
    
    5.952 nvidia-container-toolkit-base is already the newest version (1.17.8-1).
    5.952 0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.
    6.009 time="2025-07-18T09:42:39Z" level=info msg="Using /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.575.64.03"
    6.055 time="2025-07-18T09:42:39Z" level=info msg="Auto-detected mode as 'nvml'"
    6.057 time="2025-07-18T09:42:39Z" level=error msg="failed to generate CDI spec: failed to create device CDI specs: failed to initialize NVML: Driver/library version mismatch"
    ------
    ------
     > [2/2] RUN --device=nvidia.com/gpu nvidia-smi -L:
    ------
    WARNING: local cache import at .buildx-cache not found due to err: could not read .buildx-cache/index.json: open .buildx-cache/index.json: no such file or directory
    Dockerfile.test:3
    --------------------
       1 |     # syntax=docker/dockerfile:1.17-labs
       2 |     FROM ubuntu
       3 | >>> RUN --device=nvidia.com/gpu nvidia-smi -L
       4 |     
    --------------------
    ERROR: failed to build: failed to solve: failed to compute cache key: failed to create nvidia.com/gpu device: failed to generate CDI spec: exit status 1
    
  11. crazy-max commented on Jul 18, 2025

    @crazy-max
    Member

    @oisinturbittcx1 Thanks for your feedback

    Looking at the logs:

    #7 6.057 time="2025-07-18T09:42:39Z" level=error msg="failed to generate CDI spec: failed to create device CDI specs: failed to initialize NVML: Driver/library version mismatch"
    

    Seems there is a mismatch with drivers and library version. It might be that we install latest client libraries during build but drivers on your host are too old and not compatible. @tonistiigi I wonder if we should pin client libraries to an older version to increase compat?

    Maybe this could be a workaround for WSL case but mounting from the host could be error prone if a remote context/builder is used. Could maybe be an opt-in or we could detect if we are using the default context and allow it in such case.

    @pechersky Opened #3320 and seems to work on my side if you want to try.

  12. crazy-max commented on Jul 21, 2025

    @crazy-max
    Member

    @oisinturbittcx1 Documentation published: https://docs.docker.com/build/building/cdi/

    ERROR: failed to build: failed to solve: failed to load LLB: device nvidia.com/gpu=all is requested by the build but not allowed
    

    And fixes for this error here:

  13. oisinturbittcx1 commented on Jul 23, 2025

    @oisinturbittcx1
    Author

    @crazy-max Thank you! The docs are great.

    On the driver mismatch issue, I have been trying this issue on two different instances: a AWS EC2 g5.2xlarge and a LambdaLabs H100 server. Running the above setup on the H100 server works with no issues, however it is the EC2 instance that gives the driver mismatch error.

    On the H100:

    sudo docker exec -it buildx_buildkit_gpu-builde
    r0 nvidia-smi
    Wed Jul 23 15:48:01 2025       
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 570.148.08             Driver Version: 570.148.08     CUDA Version: 12.8     |
    |-----------------------------------------+------------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
    |                                         |                        |               MIG M. |
    |=========================================+========================+======================|
    |   0  NVIDIA H100 80GB HBM3          On  |   00000000:07:00.0 Off |                    0 |
    | N/A   29C    P0            117W /  700W |   76026MiB /  81559MiB |      0%      Default |
    |                                         |                        |             Disabled |
    +-----------------------------------------+------------------------+----------------------+
                                                                                             
    +-----------------------------------------------------------------------------------------+
    | Processes:                                                                              |
    |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
    |        ID   ID                                                               Usage      |
    |=========================================================================================|
    |  No running processes found                                                             |
    +-----------------------------------------------------------------------------------------+
    

    While on the A10G:

    docker exec -it buildx_buildkit_gpu-builder0 nvidia-smi   
    Failed to initialize NVML: Driver/library version mismatch
    NVML library version: 570.172
    

    What is weird is that the H100 server has a driver version of 570.148.08 while the A10G is on 570.158.01. So it appears the H100 builder is pulling in the correct system driver while the A10G builder is pulling in an incorrect one. There is nothing obvious in the CDI for why it is looking for 570.172:

    ---
    cdiVersion: 0.5.0
    containerEdits:
      deviceNodes:
      - path: /dev/nvidia-modeset
      - path: /dev/nvidia-uvm
      - path: /dev/nvidia-uvm-tools
      - path: /dev/nvidiactl
      env:
      - NVIDIA_VISIBLE_DEVICES=void
      hooks:
      - args:
        - nvidia-cdi-hook
        - create-symlinks
        - --link
        - libcuda.so.1::/usr/lib/x86_64-linux-gnu/libcuda.so
        env:
        - NVIDIA_CTK_DEBUG=false
        hookName: createContainer
        path: /usr/bin/nvidia-cdi-hook
      - args:
        - nvidia-cdi-hook
        - enable-cuda-compat
        - --host-driver-version=570.158.01
        env:
        - NVIDIA_CTK_DEBUG=false
        hookName: createContainer
        path: /usr/bin/nvidia-cdi-hook
      - args:
        - nvidia-cdi-hook
        - update-ldcache
        - --folder
        - /usr/lib/x86_64-linux-gnu
        env:
        - NVIDIA_CTK_DEBUG=false
        hookName: createContainer
        path: /usr/bin/nvidia-cdi-hook
      mounts:
      - containerPath: /run/nvidia-persistenced/socket
        hostPath: /run/nvidia-persistenced/socket
        options:
        - ro
        - nosuid
        - nodev
        - bind
        - noexec
      - containerPath: /usr/bin/nvidia-cuda-mps-control
        hostPath: /usr/bin/nvidia-cuda-mps-control
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/bin/nvidia-cuda-mps-server
        hostPath: /usr/bin/nvidia-cuda-mps-server
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/bin/nvidia-debugdump
        hostPath: /usr/bin/nvidia-debugdump
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/bin/nvidia-persistenced
        hostPath: /usr/bin/nvidia-persistenced
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/bin/nvidia-smi
        hostPath: /usr/bin/nvidia-smi
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libcuda.so.570.158.01
        hostPath: /usr/lib/x86_64-linux-gnu/libcuda.so.570.158.01
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libcudadebugger.so.570.158.01
        hostPath: /usr/lib/x86_64-linux-gnu/libcudadebugger.so.570.158.01
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.570.158.01
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.570.158.01
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.570.158.01
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.570.158.01
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.570.158.01
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.570.158.01
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.570.158.01
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.570.158.01
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.570.158.01
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.570.158.01
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.570.158.01
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-pkcs11.so.570.158.01
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.570.158.01
        hostPath: /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.570.158.01
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /lib/firmware/nvidia/570.158.01/gsp_ga10x.bin
        hostPath: /lib/firmware/nvidia/570.158.01/gsp_ga10x.bin
        options:
        - ro
        - nosuid
        - nodev
        - bind
      - containerPath: /lib/firmware/nvidia/570.158.01/gsp_tu10x.bin
        hostPath: /lib/firmware/nvidia/570.158.01/gsp_tu10x.bin
        options:
        - ro
        - nosuid
        - nodev
        - bind
    devices:
    - containerEdits:
        deviceNodes:
        - path: /dev/nvidia0
        - path: /dev/dri/card1
        - path: /dev/dri/renderD128
        hooks:
        - args:
          - nvidia-cdi-hook
          - create-symlinks
          - --link
          - ../card1::/dev/dri/by-path/pci-0000:00:1e.0-card
          - --link
          - ../renderD128::/dev/dri/by-path/pci-0000:00:1e.0-render
          env:
          - NVIDIA_CTK_DEBUG=false
          hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
        - args:
          - nvidia-cdi-hook
          - chmod
          - --mode
          - "755"
          - --path
          - /dev/dri
          env:
          - NVIDIA_CTK_DEBUG=false
          hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
      name: "0"
    - containerEdits:
        deviceNodes:
        - path: /dev/nvidia0
        - path: /dev/dri/card1
        - path: /dev/dri/renderD128
        hooks:
        - args:
          - nvidia-cdi-hook
          - create-symlinks
          - --link
          - ../card1::/dev/dri/by-path/pci-0000:00:1e.0-card
          - --link
          - ../renderD128::/dev/dri/by-path/pci-0000:00:1e.0-render
          env:
          - NVIDIA_CTK_DEBUG=false
          hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
        - args:
          - nvidia-cdi-hook
          - chmod
          - --mode
          - "755"
          - --path
          - /dev/dri
          env:
          - NVIDIA_CTK_DEBUG=false
          hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
      name: GPU-da890e30-8eaf-d47d-665f-cec65655883f
    - containerEdits:
        deviceNodes:
        - path: /dev/nvidia0
        - path: /dev/dri/card1
        - path: /dev/dri/renderD128
        hooks:
        - args:
          - nvidia-cdi-hook
          - create-symlinks
          - --link
          - ../card1::/dev/dri/by-path/pci-0000:00:1e.0-card
          - --link
          - ../renderD128::/dev/dri/by-path/pci-0000:00:1e.0-render
          env:
          - NVIDIA_CTK_DEBUG=false
          hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
        - args:
          - nvidia-cdi-hook
          - chmod
          - --mode
          - "755"
          - --path
          - /dev/dri
          env:
          - NVIDIA_CTK_DEBUG=false
          hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
      name: all
    kind: nvidia.com/gpu
    
  14. oisinturbittcx1 commented on Jul 28, 2025

    @oisinturbittcx1
    Author

    @crazy-max got to the source of the driver mismatch. The A10G server I am on is using the the nvidia-driver-XXX-server driver variants but the crazymax/buildkit:v0.23.2-ubuntu-nvidia image requires the nvidia-driver-XXX driver variants.

  15. crazy-max commented on Jul 28, 2025

    @crazy-max
    Member

    @crazy-max got to the source of the driver mismatch. The A10G server I am on is using the the nvidia-driver-XXX-server driver variants but the crazymax/buildkit:v0.23.2-ubuntu-nvidia image requires the nvidia-driver-XXX driver variants.

    Thanks for your feedback I will check this on our side.

  16. IAmIlliest commented on Aug 13, 2025

    @IAmIlliest

    FYI, I'm running into some similar issue as the OP, but haven't really got time to investigate it before giving the hardware back (this is a Jetson AGX Orin on Jetpack 6.2.1 and L4T 36.4.4).

    Following these docs:
    https://docs.docker.com/build/building/cdi/

    Am I correct in my reading of those docs, and these steps are meant to be able to run with the default builder?

    For what it's worth, on this host I can use CDI with docker run commands:
    all devices
    docker run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L
    specific devices

    docker run --rm \
        --device nvidia.com/gpu=0 \
    	ubuntu nvidia-smi -L
    	#--device nvidia.com/gpu=1:0 \
    

    Similarly, I can expose the GPU with the '--runtime=nvidia' flag for docker run as well so in general, the GPU's can be made accessible to containers for runtime. I'm needing it for build time though, similar to the OP.

    In the building CDI docs (link at the top), I get down to the point of running the buildx commands with a dockerfile called CDI_dockerfile:

    # syntax=docker/dockerfile:1-labs
    FROM busybox
    RUN --device=vendor1.com/device   env | grep ^FOO=
    

    The first buildx command fails as expected:

    docker buildx build -f ./CDI_Dockerfile ./build_context
    [+] Building 5.0s (5/5) FINISHED                                                                               docker:default
     => [internal] load build definition from CDI_Dockerfile                                                                 0.0s
     => => transferring dockerfile: 139B                                                                                     0.0s
     => resolve image config for docker-image://docker.io/docker/dockerfile:1-labs                                           2.0s
     => CACHED docker-image://docker.io/docker/dockerfile:1-labs@sha256:9187104f31e3a002a8a6a3209ea1f937fb7486c093cbbde1e14  0.0s
     => => resolve docker.io/docker/dockerfile:1-labs@sha256:9187104f31e3a002a8a6a3209ea1f937fb7486c093cbbde1e14b0fa0d7e4f1  0.0s
     => [internal] load metadata for docker.io/library/busybox:latest                                                        2.6s
     => [internal] load .dockerignore                                                                                        0.0s
     => => transferring context: 2B                                                                                          0.0s
    ERROR: failed to build: failed to solve: failed to load LLB: device vendor1.com/device=foo is requested by the build but not allowed
    

    The next step says to use the '--allow device' flag with the buildx command to fix the error above:
    (The resultant error is the same here whether I have the allow annotation added in /etc/cdi/foo.yaml or not)

    docker buildx build --allow device -f ./CDI_Dockerfile ./build_context
    [+] Building 0.0s (0/0)                                                                                        docker:default
    ERROR: failed to build: failed to solve: granting entitlement device is not allowed by build daemon configuration
    

    As you can see, the allow command is being disallowed by something in my daemon config (apparently). It's strange because the default builder can see the CDI devices. I tried explicitly enabling CDI in daemon.json along with setting the default runtime and then restarting docker.service, but the result is the same error. Here's a look at the daemon.json:

    cat /etc/docker/daemon.json
    {
        "debug": false,
        "default-runtime": "runc",
        "features": {
            "cdi": true,
            "containerd-snapshotter": true
        },
        "insecure-registries": [
            "192.168.20.170:5000"
        ],
        "runtimes": {
            "nvidia": {
                "args": [],
                "path": "nvidia-container-runtime"
            }
        }
    }
    

    And a look at the default builder:

    docker buildx inspect
    Name:          default
    Driver:        docker
    Last Activity: 2025-08-13 10:56:57 +0000 UTC
    
    Nodes:
    Name:             default
    Endpoint:         default
    Status:           running
    BuildKit version: v0.23.2
    Platforms:        linux/arm64
    Labels:
     org.mobyproject.buildkit.worker.containerd.namespace: moby
     org.mobyproject.buildkit.worker.containerd.uuid:      0d81e369-bffd-4433-99a9-51de1d096149
     org.mobyproject.buildkit.worker.executor:             containerd
     org.mobyproject.buildkit.worker.hostname:             ai-jetson
     org.mobyproject.buildkit.worker.moby.host-gateway-ip: 172.17.0.1
     org.mobyproject.buildkit.worker.network:              host
     org.mobyproject.buildkit.worker.selinux.enabled:      false
     org.mobyproject.buildkit.worker.snapshotter:          overlayfs
    Devices:
     Name:                  nvidia.com/gpu=0
     Automatically allowed: false
     Name:                  nvidia.com/gpu=all
     Automatically allowed: false
     Name:                  vendor1.com/device=foo
     Automatically allowed: false
    GC Policy rule#0:
     All:            false
     Filters:        type==source.local,type==exec.cachemount,type==source.git.checkout
     Keep Duration:  48h0m0s
     Max Used Space: 790.6MiB
    GC Policy rule#1:
     All:            false
     Keep Duration:  1440h0m0s
     Reserved Space: 5.588GiB
     Max Used Space: 953.7MiB
     Min Free Space: 2.794GiB
    GC Policy rule#2:
     All:            false
     Reserved Space: 5.588GiB
     Max Used Space: 953.7MiB
     Min Free Space: 2.794GiB
    GC Policy rule#3:
     All:            true
     Reserved Space: 5.588GiB
     Max Used Space: 953.7MiB
     Min Free Space: 2.794GiB
    

    Same error results even with the allow annotations which you can see is working per the builder:

    docker buildx inspect
    Name:          default
    Driver:        docker
    Last Activity: 2025-08-13 11:03:11 +0000 UTC
    
    Nodes:
    Name:             default
    Endpoint:         default
    Status:           running
    BuildKit version: v0.23.2
    Platforms:        linux/arm64
    Labels:
     org.mobyproject.buildkit.worker.containerd.namespace: moby
     org.mobyproject.buildkit.worker.containerd.uuid:      0d81e369-bffd-4433-99a9-51de1d096149
     org.mobyproject.buildkit.worker.executor:             containerd
     org.mobyproject.buildkit.worker.hostname:             ai-jetson
     org.mobyproject.buildkit.worker.moby.host-gateway-ip: 172.17.0.1
     org.mobyproject.buildkit.worker.network:              host
     org.mobyproject.buildkit.worker.selinux.enabled:      false
     org.mobyproject.buildkit.worker.snapshotter:          overlayfs
    Devices:
     Name:                  nvidia.com/gpu=0
     Automatically allowed: false
     Name:                  nvidia.com/gpu=all
     Automatically allowed: false
     Name:                  vendor1.com/device=foo
     Automatically allowed: true
     Annotations:
      org.mobyproject.buildkit.device.autoallow: true
    GC Policy rule#0:
     All:            false
     Filters:        type==source.local,type==exec.cachemount,type==source.git.checkout
     Keep Duration:  48h0m0s
     Max Used Space: 790.6MiB
    GC Policy rule#1:
     All:            false
     Keep Duration:  1440h0m0s
     Reserved Space: 5.588GiB
     Max Used Space: 953.7MiB
     Min Free Space: 2.794GiB
    GC Policy rule#2:
     All:            false
     Reserved Space: 5.588GiB
     Max Used Space: 953.7MiB
     Min Free Space: 2.794GiB
    GC Policy rule#3:
     All:            true
     Reserved Space: 5.588GiB
     Max Used Space: 953.7MiB
     Min Free Space: 2.794GiB
    

    Any ideas if I'm doing something wrong here?

  17. oisinturbittcx1 commented on Aug 14, 2025

    @oisinturbittcx1
    Author

    Am I correct in my reading of those docs, and these steps are meant to be able to run with the default builder?

    I have to do the following:

    1. Setup the [CDI Specification](https://docs.docker.com/reference/cli/dockerd/#configure-cdi-devices) using [Nvidia Container Toolkit's CDI support](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html):
    
        ```bash
        sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
        sudo systemctl restart docker
        ```
    
        You may need to add the ensure the CDI spec has the following:
    
        ```yaml
        cdiVersion: "0.6.0"
        ...
        annotations:
            org.mobyproject.buildkit.device.autoallow: true
        ```
    
    2. Create a new [Buildx](https://github.com/docker/buildx) builder that allows usage of the instance's GPU during the build stage:
    
        ```bash
        docker buildx create --name gpu-builder --driver-opt "image=crazymax/buildkit:v0.23.2-ubuntu-nvidia" --bootstrap --use
        ```
    
    3. Verify the new builder workers by running: `docker exec -it buildx_buildkit_gpu-builder0 nvidia-smi`
    
  18. IAmIlliest commented on Aug 15, 2025

    @IAmIlliest

    Thanks for the clarification. I had gathered from this discussion that only worked with the new GPU builder. My confusion arose because the official docs seemed to imply it worked with the default builder....nevertheless, gpu-builder to the rescue.

  19. crazy-max commented on Aug 22, 2025

    @crazy-max
    Member

    @IAmIlliest Yes this will be solved on next BuildKit release: #3295 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions