Skip to content

[Bug]: a partially specified NVIDIADriver startupProbe renders zeros instead of operator defaults #3132

Open

Description

@blacksoxx

Describe the bug

When an NVIDIADriver CR sets spec.startupProbe but leaves some fields unset, the GPU Operator renders explicit zeros for those fields into the driver DaemonSet, so the driver startup window collapses to the generic Kubernetes probe defaults instead of the operator's intended values.

getDriverSpec (internal/state/driver.go) applies getDefaultStartupProbe only when spec.StartupProbe == nil:

if spec.StartupProbe == nil {
    spec.StartupProbe = getDefaultStartupProbe(spec)
}

If the CR specifies the probe partially (for example startupProbe: {initialDelaySeconds: 120}), the unset fields stay zero, and manifests/state-driver/0500_daemonset.yaml renders failureThreshold: 0, periodSeconds: 0, timeoutSeconds: 0, successThreshold: 0. The API server then substitutes the generic defaults (failureThreshold 3, periodSeconds 10, timeoutSeconds 1) rather than the operator defaults (failureThreshold 120, timeoutSeconds 60). A driver that compiles for more than ~30s, or whose nvidia-smi probe takes longer than 1s, is then killed and restarted in a loop.

The ClusterPolicy path does not have this problem: setContainerProbe (controllers/object_controls.go) overlays only the non-zero user fields on top of the defaults. The two paths disagree.

To Reproduce

Apply an NVIDIADriver CR whose spec.startupProbe sets only initialDelaySeconds (any single field), and inspect the rendered driver DaemonSet: the other probe fields are 0.

Reproduced with a unit test over getDriverSpec: a CR with StartupProbe{InitialDelaySeconds: 120} yields failureThreshold 0 / periodSeconds 0 / timeoutSeconds 0 / successThreshold 0 instead of the operator defaults.

Expected behavior

A partially specified startupProbe keeps the fields the user set and falls back to the operator defaults for the rest, as the ClusterPolicy path already does.

Environment (please provide the following information):

  • GPU Operator Version: present in v26.7.2 and on main (internal/state/driver.go has no overlay for a partial probe); reproduced on main
  • OS: N/A for the reproduction (unit test, macOS arm64, Go 1.26)
  • Kernel Version: N/A (not run on a cluster)
  • Container Runtime Version: N/A (not run on a cluster)
  • Kubernetes Distro and Version: N/A (not run on a cluster; reproduced with the internal/state unit tests)

A fix is in the linked PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions