You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(driver): default the unset fields of a partial startupProbe - #3133
getDriverSpec applied the operator's default startupProbe only when spec.StartupProbe was nil. A partially specified probe (for example only initialDelaySeconds) left the other fields at zero, so the driver DaemonSet rendered failureThreshold: 0 / periodSeconds: 0 / timeoutSeconds: 0 and the API server substituted the generic Kubernetes defaults (failureThreshold 3, periodSeconds 10, timeoutSeconds 1) instead of the operator's longer startup window — which can restart-loop a driver that is still compiling.
This adds applyStartupProbeDefaults, which fills the unset (zero) fields of a user-specified probe from the operator defaults, matching setContainerProbe on the ClusterPolicy path so the two paths agree.
Checklist
No secrets, sensitive information, or unrelated changes
Lint checks passing (make lint) — see Testing note
Generated assets in-sync (make validate-generated-assets) — no API type changes
Go mod artifacts in-sync (make validate-modules) — no dependency changes
Test cases are added for new code paths
Testing
Added TestGetDriverSpecStartupProbeDefaults (table-driven) in internal/state/driver_sync_test.go: a nil probe still gets all operator defaults, and a partial probe (initialDelaySeconds: 120) keeps that value while defaulting the rest. I confirmed the partial case fails without the fix (renders timeoutSeconds 0 / failureThreshold 0).
Ran locally (Go 1.26, macOS arm64): gofmt -s clean, go vet ./internal/state/, and go test ./internal/state/ all pass. make lint could not run locally — the golangci-lint build available to me is compiled with go1.26 and panics on a dependency that requires go1.27; leaving the full lint to CI.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Explicit zero startup delay is lost
When a driver specifies initialDelaySeconds: 0, applyStartupProbeDefaults replaces it with 60 seconds, or 5 for precompiled drivers. The first startup check runs late despite the valid zero-delay setting.
Learn more
The NVIDIADriver probe fields are int32 values with omitempty, so the decoded Go value cannot distinguish an omitted delay from an explicit zero. Kubernetes accepts zero as an initial delay, and the DaemonSet template previously rendered that zero directly. Filling every zero delay therefore changes behavior for existing resources with initialDelaySeconds: 0.
Example: A driver with startupProbe: {initialDelaySeconds: 0, failureThreshold: 120} previously started probing immediately. After this change, a standard driver waits 60 seconds before its first probe.
Recommended fix: Preserve an explicitly supplied zero delay while still defaulting an omitted delay. This requires tracking field presence (for example with a pointer field and corresponding CRD/deepcopy updates) or selecting an omission behavior that does not silently reinterpret a user's zero. Add a regression test using an explicit zero in a CR and verify the rendered DaemonSet probe.
Was this helpful? React with 👍 or 👎 to provide feedback.
The reason will be displayed to describe this comment to others. Learn more.
Thanks - this is deliberate and matches the existing ClusterPolicy path: setContainerProbe (controllers/object_controls.go) also treats a zero field as unset and applies the default. ContainerProbeSpec fields are int32, not pointers, so an explicit 0 is indistinguishable from unset; I kept the two paths consistent rather than diverging one. Distinguishing an explicit zero delay would be a separate change to both paths (e.g. pointer fields).
When an NVIDIADriver startup probe is partially specified, getDriverSpec now fills each zero-valued field with the corresponding operator default. Configured nonzero values remain unchanged. Tests cover a nil probe and a partial probe with InitialDelaySeconds set to 120.
Priority: Normal
Severity of issue fixed: Medium
Merge Risk: Minimal · up to bb4bd
Partial probes receive operator defaults while retaining configured values. The remaining test recommendation does not block merging.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
Cover non-default values in the partial-probe case.
TestGetDriverSpecPreservesUserStartupProbe uses a fully populated probe. It does not test merging a partial probe. Add non-default TimeoutSeconds, PeriodSeconds, and FailureThreshold values to a partial-probe case and assert that getDriverSpec preserves them.
Reviewing files that changed from the base of the PR and between ff1e017 and bb4bd1f.
Files selected for processing: 2
internal/state/driver.go
internal/state/driver_sync_test.go
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
getDriverSpec applied the operator's default startupProbe only when the whole
spec.StartupProbe was nil. A partially specified startupProbe (for example only
initialDelaySeconds) rendered explicit zeros into the driver DaemonSet for the
fields left unset, so the API server substituted the generic Kubernetes probe
defaults (failureThreshold 3, periodSeconds 10, timeoutSeconds 1) in place of the
operator's longer driver-startup window, which can restart-loop a driver that is
still compiling. Fill the unset fields from the defaults, matching
setContainerProbe on the ClusterPolicy path.
Signed-off-by: Youssef Omar Bouden <youssef.bouden2002@gmail.com>
Thanks @abrarshivani - rebased onto the latest main (now at 5e4dd6ff), clean with no conflicts, and the internal/state startupProbe test still passes locally.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #3132.
getDriverSpecapplied the operator's defaultstartupProbeonly whenspec.StartupProbewasnil. A partially specified probe (for example onlyinitialDelaySeconds) left the other fields at zero, so the driver DaemonSet renderedfailureThreshold: 0/periodSeconds: 0/timeoutSeconds: 0and the API server substituted the generic Kubernetes defaults (failureThreshold 3, periodSeconds 10, timeoutSeconds 1) instead of the operator's longer startup window — which can restart-loop a driver that is still compiling.This adds
applyStartupProbeDefaults, which fills the unset (zero) fields of a user-specified probe from the operator defaults, matchingsetContainerProbeon the ClusterPolicy path so the two paths agree.Checklist
make lint) — see Testing notemake validate-generated-assets) — no API type changesmake validate-modules) — no dependency changesTesting
Added
TestGetDriverSpecStartupProbeDefaults(table-driven) ininternal/state/driver_sync_test.go: a nil probe still gets all operator defaults, and a partial probe (initialDelaySeconds: 120) keeps that value while defaulting the rest. I confirmed the partial case fails without the fix (renderstimeoutSeconds 0 / failureThreshold 0).Ran locally (Go 1.26, macOS arm64):
gofmt -sclean,go vet ./internal/state/, andgo test ./internal/state/all pass.make lintcould not run locally — the golangci-lint build available to me is compiled with go1.26 and panics on a dependency that requires go1.27; leaving the full lint to CI.