Skip to content

[Bug]: v26.7.x labels sandbox nodes with the container workload despite `sandboxWorkloads.defaultWorkload=vm-passthrough #3131

Open

Description

@Miaoxiang-philips

Describe the bug

With sandboxWorkloads.enabled=true and sandboxWorkloads.defaultWorkload=vm-passthrough, GPU Operator v26.7.x can label a GPU node without a nvidia.com/gpu.workload.config label with the container operand labels instead of the vm-passthrough ones. The VFIO manager and sandbox device plugin are then not deployed, and KubeVirt GPU passthrough fails. The wrong labels persist until the operator is restarted after a manual node label is added.

The operator log shows sandbox workloads enabled, but the default workload is still container:

WARNING: failed to get GPU workload config for node; using default
  NodeName=<node> SandboxEnabled=true Error="no GPU workload config found"
  defaultGPUWorkloadConfig=container

This configuration is documented as supported in GPU Operator with KubeVirt: "To override the default GPU workload configuration, set the following value in ClusterPolicy: sandboxWorkloads.defaultWorkload=<config>." It worked reliably for us with v25.3.1.

Analysis

In v26.7.0, node labeling moved from ClusterPolicyController.init() into the separate NodeLabelingReconciler. Both controllers react to ClusterPolicy creation and run concurrently, but:

  • NodeLabelingReconciler reads sandboxEnabled directly from the ClusterPolicy spec (controllers/nodelabeling_controller.go, updateGPUStateLabels).
  • For nodes without the workload label, it falls back to the package-level defaultGPUWorkloadConfig (controllers/state_manager.go, getWorkloadConfig). This variable is initialized to container.
  • That variable is updated from spec.sandboxWorkloads.defaultWorkload only in ClusterPolicyController.init(), which is called by ClusterPolicyReconciler after several API calls.

If node labeling runs first, it applies container operand labels. Nothing corrects this afterward:

  • The variable assignment is not a Kubernetes event.
  • Node update events are filtered by nodeUpdateRequiresReconcile.
  • ClusterPolicy update events are generation-only.
  • There is no periodic resync.

The same race recurs every time the operator restarts, because the variable is reset to container. That can also flip nodes that were previously labeled correctly.

Through v26.3.3, labelGPUNodes() ran inside ClusterPolicyController.init() after defaultGPUWorkloadConfig was assigned, so this ordering was guaranteed.

To Reproduce

  1. Use a single-node cluster with an NVIDIA GPU and NFD providing PCI labels, and no nvidia.com/gpu.workload.config node label.

  2. Install GPU Operator v26.7.2:

    sandboxWorkloads:
      enabled: true
      defaultWorkload: vm-passthrough
    driver:
      enabled: false
    toolkit:
      enabled: false
  3. Inspect the node labels and operator logs. On affected installs, the node receives nvidia.com/gpu.deploy.device-plugin=true and the other container operand labels instead of nvidia.com/gpu.deploy.vfio-manager=true and nvidia.com/gpu.deploy.sandbox-device-plugin=true. The log contains defaultGPUWorkloadConfig=container.

The issue is timing-dependent, so it may not reproduce on every install. Restarting the operator pod repeats the race.

Expected behavior

Nodes without nvidia.com/gpu.workload.config always use spec.sandboxWorkloads.defaultWorkload from the ClusterPolicy, independent of controller ordering or operator restarts.

A possible fix: in NodeLabelingReconciler, derive the fallback workload from cp.Spec.SandboxWorkloads.DefaultWorkload (when it is valid) instead of the shared package-level variable. The same applies to getNodeLabelUpdateReasons.

Environment (please provide the following information):

  • GPU Operator Version: v26.7.2 (also present in v26.7.0 and v26.7.1 by code inspection; not present in v26.3.3 or earlier)
  • OS: SUSE Linux Micro 6.1
  • Kernel Version: 6.4.0 (SUSE)
  • Container Runtime Version: containerd v2.x (bundled with K3s)
  • Kubernetes Distro and Version: K3s v1.36.5+k3s1
  • Workload: KubeVirt VMs with GPU passthrough; NFD v0.20 installed separately (nfd.enabled=false)

Workaround

Explicitly label the node with nvidia.com/gpu.workload.config=vm-passthrough and restart the operator pod, or use v26.3.3 or earlier.

Activity

  1. added
    bugIssue/PR to expose/discuss/fix a bug
    needs-triageissue or PR has not been assigned a priority-px label
    on Oct 28, 2026
  2. self-assigned this
    on Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugIssue/PR to expose/discuss/fix a bugneeds-triageissue or PR has not been assigned a priority-px label

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions