Skip to content

[Bug]: GPUCluster DRA kubelet plugin uses driver root / on Talos and cannot find libnvidia-ml.so.1 #3139

Open

Description

@Moep90

Describe the bug

On Talos Linux the DRA kubelet plugin deployed through a GPUCluster crashes with error locating "libnvidia-ml.so.1", so no ResourceSlice is published and DCGM Exporter and the DRA validation stay Pending.

Talos ships the driver as a system extension under /usr/local, and /usr/bin/nvidia-smi is a link to /usr/local/bin/nvidia-smi. The validator's runValidation tries the host driver first: resolveHostNvidiaSMI finds /usr/bin/nvidia-smi, chroot /host /usr/bin/nvidia-smi succeeds, and getDriverInfo(true, ...) sets the driver root to the host root. driver-ready then contains NVIDIA_DRIVER_ROOT=/ and hostPaths.driverInstallDir: /usr/local from the GPUCluster is never used. The kubelet plugin sources driver-ready (manifests/state-dra-driver/0500_daemonset.yaml), looks for libnvidia-ml.so.1 below / and does not find it in /usr/local/lib.

I found no field in GPUCluster to set the driver root. Setting it through kubeletPlugin.env has no effect, because the container sources driver-ready afterwards and overwrites it.

As a check I patched NVIDIA_DRIVER_ROOT=/usr/local into the kubelet plugin DaemonSet by hand. With that the GPUCluster becomes ready and DCGM Exporter labels the metrics with the pod holding the DRA claim, so the rest of the path works on Talos:

DCGM_FI_DEV_GPU_UTIL{gpu="0",...,container="cuda",dra_claim_name="dra-hold-gpu-hj5r2",dra_claim_namespace="default",dra_device_name="gpu-0",dra_driver_name="gpu.nvidia.com",dra_pool_name="gpu",namespace="default",pod="dra-hold"} 0

The standalone DRA driver chart works on the same node with nvidiaDriverRoot: /usr/local.

To Reproduce

  1. Talos node with the NVIDIA system extensions (driver under /usr/local).
  2. Install the GPU Operator v26.7.1 with driver and toolkit disabled, as the Talos guide describes, and create a GPUCluster with the DRA kubelet plugin enabled and hostPaths.driverInstallDir: /usr/local.
  3. The kubelet plugin pod crash-loops with error locating "libnvidia-ml.so.1"; /run/nvidia/validations/driver-ready on the node contains IS_HOST_DRIVER=true and NVIDIA_DRIVER_ROOT=/.

Expected behavior

The driver root the plugin uses is /usr/local, either because a host driver found under a non-standard prefix is reported with that prefix, or because an explicit hostPaths.driverInstallDir (or a dedicated field) wins over the host-driver detection.

Environment

  • GPU Operator Version: v26.7.1 (I read v26.7.2 and main and see the same logic, not tested live)
  • DRA driver: v0.5.1, DCGM Exporter 4.6.1-4.8.4-distroless
  • OS: Talos Linux v1.14.5 with the NVIDIA proprietary driver extensions
  • Container Runtime Version: containerd as shipped with Talos v1.14.5
  • Kubernetes Distro and Version: upstream Kubernetes 1.37.1
  • GPU: RTX A1000

Related: #2731

Activity

  1. roma5087 commented on Oct 29, 2026

    @roma5087

    I'd like to work on this. Can this be assigned it to me?

    Proposed fix:

    1. In the host-driver branch of getDriverInfo, if the mounted driverInstallDir actually contains nvidia-smi and libnvidia-ml.so.1, report it as NVIDIA_DRIVER_ROOT. Otherwise keep today's / and /host. On a standard host-driver node /run/nvidia/driver is an empty DirectoryOrCreate mount, so driver-ready stays byte-identical there.

    2. Add /lib and /usr/lib to the library search paths in driver.Root.

    3. Add regression tests. One shows the default host-driver output is unchanged, since ClusterPolicy's device-plugin, toolkit and mig-manager also source driver-ready. The other covers a /usr/local-style layout.

    Two questions before I start:

    1. Would you prefer an explicit field (for example hostPaths.driverRoot, matching the standalone DRA chart's nvidiaDriverRoot) over detecting it from driverInstallDir? I'd only go that way with your sign-off, since it's an API addition.

    2. fix(validator): re-probe the host driver while waiting for a driver container #3034 also changes runValidation and adds a second host-driver return path. I'd put the change inside getDriverInfo so it covers both paths. Is it fine to base this on fix(validator): re-probe the host driver while waiting for a driver container #3034 if that lands first?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions