Skip to content

fix(driver): default the unset fields of a partial startupProbe - #3133

Open
blacksoxx wants to merge 1 commit into
NVIDIA:mainfrom
blacksoxx:fix-driver-partial-startupprobe
Open

blacksoxx wants to merge 1 commit into
NVIDIA:mainfrom
blacksoxx:fix-driver-partial-startupprobe

Conversation

@blacksoxx

@blacksoxx blacksoxx commented Oct 28, 2026 •

Copy link
Copy Markdown

Description

Fixes #3132.

getDriverSpec applied the operator's default startupProbe only when spec.StartupProbe was nil. A partially specified probe (for example only initialDelaySeconds) left the other fields at zero, so the driver DaemonSet rendered failureThreshold: 0 / periodSeconds: 0 / timeoutSeconds: 0 and the API server substituted the generic Kubernetes defaults (failureThreshold 3, periodSeconds 10, timeoutSeconds 1) instead of the operator's longer startup window — which can restart-loop a driver that is still compiling.

This adds applyStartupProbeDefaults, which fills the unset (zero) fields of a user-specified probe from the operator defaults, matching setContainerProbe on the ClusterPolicy path so the two paths agree.

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Lint checks passing (make lint) — see Testing note
  • Generated assets in-sync (make validate-generated-assets) — no API type changes
  • Go mod artifacts in-sync (make validate-modules) — no dependency changes
  • Test cases are added for new code paths

Testing

Added TestGetDriverSpecStartupProbeDefaults (table-driven) in internal/state/driver_sync_test.go: a nil probe still gets all operator defaults, and a partial probe (initialDelaySeconds: 120) keeps that value while defaulting the rest. I confirmed the partial case fails without the fix (renders timeoutSeconds 0 / failureThreshold 0).

Ran locally (Go 1.26, macOS arm64): gofmt -s clean, go vet ./internal/state/, and go test ./internal/state/ all pass. make lint could not run locally — the golangci-lint build available to me is compiled with go1.26 and panics on a dependency that requires go1.27; leaving the full lint to CI.


Devin Review

@blacksoxx
blacksoxx requested a review from a team as a code owner October 28, 2026 11:31
@copy-pr-bot

copy-pr-bot Bot commented Oct 28, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Devin Review

Comment thread internal/state/driver.go
Comment on lines +557 to +559
if probe.InitialDelaySeconds == 0 {
probe.InitialDelaySeconds = defaults.InitialDelaySeconds
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Explicit zero startup delay is lost

When a driver specifies initialDelaySeconds: 0, applyStartupProbeDefaults replaces it with 60 seconds, or 5 for precompiled drivers. The first startup check runs late despite the valid zero-delay setting.

Learn more

The NVIDIADriver probe fields are int32 values with omitempty, so the decoded Go value cannot distinguish an omitted delay from an explicit zero. Kubernetes accepts zero as an initial delay, and the DaemonSet template previously rendered that zero directly. Filling every zero delay therefore changes behavior for existing resources with initialDelaySeconds: 0.

Example: A driver with startupProbe: {initialDelaySeconds: 0, failureThreshold: 120} previously started probing immediately. After this change, a standard driver waits 60 seconds before its first probe.

Recommended fix: Preserve an explicitly supplied zero delay while still defaulting an omitted delay. This requires tracking field presence (for example with a pointer field and corresponding CRD/deepcopy updates) or selecting an omission behavior that does not silently reinterpret a user's zero. Add a regression test using an explicit zero in a CR and verify the rendered DaemonSet probe.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - this is deliberate and matches the existing ClusterPolicy path: setContainerProbe (controllers/object_controls.go) also treats a zero field as unset and applies the default. ContainerProbeSpec fields are int32, not pointers, so an explicit 0 is indistinguishable from unset; I kept the two paths consistent rather than diverging one. Distinguishing an explicit zero delay would be a separate change to both paths (e.g. pointer fields).

@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Walkthrough

When an NVIDIADriver startup probe is partially specified, getDriverSpec now fills each zero-valued field with the corresponding operator default. Configured nonzero values remain unchanged. Tests cover a nil probe and a partial probe with InitialDelaySeconds set to 120.

Priority: Normal

Severity of issue fixed: Medium

Merge Risk: Minimal · up to bb4bd

Partial probes receive operator defaults while retaining configured values. The remaining test recommendation does not block merging.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick comments: 1
internal/state/driver_sync_test.go: 1

450-454: Trivial · Maintainability & Code Quality · Quick win

Cover non-default values in the partial-probe case.

TestGetDriverSpecPreservesUserStartupProbe uses a fully populated probe. It does not test merging a partial probe. Add non-default TimeoutSeconds, PeriodSeconds, and FailureThreshold values to a partial-probe case and assert that getDriverSpec preserves them.

Suggested test case
 		"partial startup probe keeps set fields and defaults the rest": {
 			startupProbe: &nvidiav1alpha1.ContainerProbeSpec{InitialDelaySeconds: 120},
 			expected: &nvidiav1alpha1.ContainerProbeSpec{
 				InitialDelaySeconds: 120, TimeoutSeconds: 60, PeriodSeconds: 10, SuccessThreshold: 1, FailureThreshold: 120,
 			},
 		},
+		"partial startup probe keeps user-set timeout, period, and failure threshold": {
+			startupProbe: &nvidiav1alpha1.ContainerProbeSpec{TimeoutSeconds: 5, PeriodSeconds: 20, FailureThreshold: 30},
+			expected: &nvidiav1alpha1.ContainerProbeSpec{
+				InitialDelaySeconds: 60, TimeoutSeconds: 5, PeriodSeconds: 20, SuccessThreshold: 1, FailureThreshold: 30,
+			},
+		},

Review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between ff1e017 and bb4bd1f.

Files selected for processing: 2
  • internal/state/driver.go
  • internal/state/driver_sync_test.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@abrarshivani

Copy link
Copy Markdown
Contributor

@blacksoxx Can you rebase to latest?

getDriverSpec applied the operator's default startupProbe only when the whole
spec.StartupProbe was nil. A partially specified startupProbe (for example only
initialDelaySeconds) rendered explicit zeros into the driver DaemonSet for the
fields left unset, so the API server substituted the generic Kubernetes probe
defaults (failureThreshold 3, periodSeconds 10, timeoutSeconds 1) in place of the
operator's longer driver-startup window, which can restart-loop a driver that is
still compiling. Fill the unset fields from the defaults, matching
setContainerProbe on the ClusterPolicy path.

Signed-off-by: Youssef Omar Bouden <youssef.bouden2002@gmail.com>
@blacksoxx
blacksoxx force-pushed the fix-driver-partial-startupprobe branch from bb4bd1f to 9e4eee0 Compare October 30, 2026 12:14
@blacksoxx

Copy link
Copy Markdown
Author

Thanks @abrarshivani - rebased onto the latest main (now at 5e4dd6ff), clean with no conflicts, and the internal/state startupProbe test still passes locally.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: a partially specified NVIDIADriver startupProbe renders zeros instead of operator defaults

2 participants