Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions internal/state/driver.go
Original file line number Diff line number Diff line change
Expand Up @@ -548,6 +548,29 @@ func getDefaultStartupProbe(spec *nvidiav1alpha1.NVIDIADriverSpec) *nvidiav1alph
}
}

// applyStartupProbeDefaults fills unset (zero) fields of a user-specified startup probe with the operator
// defaults. A partially specified startupProbe otherwise renders explicit zeros into the driver DaemonSet,
// and the API server then substitutes the generic Kubernetes probe defaults (failureThreshold 3,
// periodSeconds 10, timeoutSeconds 1) in place of the operator's longer driver-startup window, which can
// restart-loop a driver that is still compiling. This mirrors setContainerProbe on the ClusterPolicy path.
func applyStartupProbeDefaults(probe, defaults *nvidiav1alpha1.ContainerProbeSpec) {
if probe.InitialDelaySeconds == 0 {
probe.InitialDelaySeconds = defaults.InitialDelaySeconds
}
Comment on lines +557 to +559

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Explicit zero startup delay is lost

When a driver specifies initialDelaySeconds: 0, applyStartupProbeDefaults replaces it with 60 seconds, or 5 for precompiled drivers. The first startup check runs late despite the valid zero-delay setting.

Learn more

The NVIDIADriver probe fields are int32 values with omitempty, so the decoded Go value cannot distinguish an omitted delay from an explicit zero. Kubernetes accepts zero as an initial delay, and the DaemonSet template previously rendered that zero directly. Filling every zero delay therefore changes behavior for existing resources with initialDelaySeconds: 0.

Example: A driver with startupProbe: {initialDelaySeconds: 0, failureThreshold: 120} previously started probing immediately. After this change, a standard driver waits 60 seconds before its first probe.

Recommended fix: Preserve an explicitly supplied zero delay while still defaulting an omitted delay. This requires tracking field presence (for example with a pointer field and corresponding CRD/deepcopy updates) or selecting an omission behavior that does not silently reinterpret a user's zero. Add a regression test using an explicit zero in a CR and verify the rendered DaemonSet probe.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - this is deliberate and matches the existing ClusterPolicy path: setContainerProbe (controllers/object_controls.go) also treats a zero field as unset and applies the default. ContainerProbeSpec fields are int32, not pointers, so an explicit 0 is indistinguishable from unset; I kept the two paths consistent rather than diverging one. Distinguishing an explicit zero delay would be a separate change to both paths (e.g. pointer fields).

if probe.TimeoutSeconds == 0 {
probe.TimeoutSeconds = defaults.TimeoutSeconds
}
if probe.PeriodSeconds == 0 {
probe.PeriodSeconds = defaults.PeriodSeconds
}
if probe.SuccessThreshold == 0 {
probe.SuccessThreshold = defaults.SuccessThreshold
}
if probe.FailureThreshold == 0 {
probe.FailureThreshold = defaults.FailureThreshold
}
}

func getDriverImagePath(spec *nvidiav1alpha1.NVIDIADriverSpec, nodePool nodePool) (string, error) {
os := nodePool.osTag

Expand Down Expand Up @@ -595,6 +618,8 @@ func getDriverSpec(cr *nvidiav1alpha1.NVIDIADriver, nodePool nodePool) (*driverS

if spec.StartupProbe == nil {
spec.StartupProbe = getDefaultStartupProbe(spec)
} else {
applyStartupProbeDefaults(spec.StartupProbe, getDefaultStartupProbe(spec))
}

spec.Labels = sanitizeDriverLabels(spec.Labels)
Expand Down
46 changes: 46 additions & 0 deletions internal/state/driver_sync_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -418,6 +418,52 @@ func TestGetDriverSpecErrors(t *testing.T) {
})
}

func TestGetDriverSpecStartupProbeDefaults(t *testing.T) {
newDriverCR := func(probe *nvidiav1alpha1.ContainerProbeSpec) *nvidiav1alpha1.NVIDIADriver {
return &nvidiav1alpha1.NVIDIADriver{
ObjectMeta: metav1.ObjectMeta{Name: "driver-a"},
Spec: nvidiav1alpha1.NVIDIADriverSpec{
DriverType: nvidiav1alpha1.GPU,
Repository: "nvcr.io/nvidia",
Image: "driver",
Version: "535.104.05",
Manager: nvidiav1alpha1.DriverManagerSpec{
Repository: "nvcr.io/nvidia/cloud-native",
Image: "k8s-driver-manager",
Version: "v0.6.0",
},
StartupProbe: probe,
},
}
}

testCases := map[string]struct {
startupProbe *nvidiav1alpha1.ContainerProbeSpec
expected *nvidiav1alpha1.ContainerProbeSpec
}{
"nil startup probe uses all operator defaults": {
startupProbe: nil,
expected: &nvidiav1alpha1.ContainerProbeSpec{
InitialDelaySeconds: 60, TimeoutSeconds: 60, PeriodSeconds: 10, SuccessThreshold: 1, FailureThreshold: 120,
},
},
"partial startup probe keeps set fields and defaults the rest": {
startupProbe: &nvidiav1alpha1.ContainerProbeSpec{InitialDelaySeconds: 120},
expected: &nvidiav1alpha1.ContainerProbeSpec{
InitialDelaySeconds: 120, TimeoutSeconds: 60, PeriodSeconds: 10, SuccessThreshold: 1, FailureThreshold: 120,
},
},
}

for name, tc := range testCases {
t.Run(name, func(t *testing.T) {
spec, err := getDriverSpec(newDriverCR(tc.startupProbe), nodePool{osTag: "ubuntu22.04"})
require.NoError(t, err)
assert.Equal(t, tc.expected, spec.Spec.StartupProbe)
})
}
}

func TestHandleDefaultImagesInObjectsManagerImageSet(t *testing.T) {
scheme := driverTestScheme(t)
driverState := newTestStateDriver(t, newDriverFakeClient(scheme), scheme)
Expand Down