You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[gpu-op] NVIDIADriver does not schedule a retry when its secretEnv Secret is missing #3208
When a NVIDIADriver references a secretEnv Secret that does not exist yet, reconciliation records error conditions but returns an empty result and a nil error. No retry is scheduled, and the controller has no watch for the referenced Secret.
This can occur when GitOps creates the NVIDIADriver before another controller creates its Secret.
Steps to reproduce
Create a NVIDIADriver with spec.secretEnv referencing a Secret that does not exist in the GPU Operator namespace:
spec:
secretEnv: driver-env
Allow reconciliation to run and record the missing-Secret error.
Create the referenced Secret in the GPU Operator namespace.
Expected behavior
The controller retries reconciliation and proceeds once the Secret exists, without requiring an unrelated event.
While the dependency is missing, status.state should be notReady, consistent with the error conditions.
Actual behavior
The Secret lookup failure returns:
return reconcile.Result{}, nil
Creating the Secret alone does not enqueue the NVIDIADriver. Recovery depends on another qualifying event, such as a driver spec change, a relevant node-label change, or an operator restart.
Additionally, if the driver was previously ready, this branch can leave contradictory status:
Set the driver's state to notReady and return the Secret lookup error so the controller's configured retry backoff applies.
Alternatively, handle NotFound with a bounded RequeueAfter and return other lookup errors normally. A watch for referenced Secrets could provide faster recovery but is not required for a minimal fix.
I'd be interested in contributing a fix for this issue.
Before starting implementation, I wanted to clarify whether the project permits AI-assisted development using tools such as Antigravity and ChatGPT.
I intend to use these tools for codebase exploration, implementation assistance, and debugging, while personally reviewing, understanding, testing, and taking full responsibility for every change.
I will also follow the project's contribution guidelines and DCO sign-off requirements.
Would this approach be acceptable for a contribution to GPU Operator?
While investigating and working on this issue, I discovered that PR #3045 already addresses the underlying problem with a more comprehensive event-driven approach.
I initially implemented a fix independently, but after comparing both implementations, I found that PR #3045 also handles Secret creation/deletion events, avoids caching the full Secret payload, and handles transient API errors.
I also identified an opportunity to strengthen its test coverage. I've added a test that uses the real condition updater and verifies that the NotReady status and associated conditions are actually persisted.
I've reached out to the author of PR #3045 to offer this additional test.
Would you recommend contributing the test to the existing PR and allowing it to resolve this issue, or is there any additional work you'd like me to take up independently?
Activity
Wizard52007 commentedon Nov 10, 2026
Hi!
I'd be interested in contributing a fix for this issue.
Before starting implementation, I wanted to clarify whether the project permits AI-assisted development using tools such as Antigravity and ChatGPT.
I intend to use these tools for codebase exploration, implementation assistance, and debugging, while personally reviewing, understanding, testing, and taking full responsibility for every change.
I will also follow the project's contribution guidelines and DCO sign-off requirements.
Would this approach be acceptable for a contribution to GPU Operator?
Also, is anyone currently working on this issue?
Thanks!
rahulait commentedon Nov 10, 2026
@Wizard52007 , yes, that sounds ok. I'll assign the issue to you.
Wizard52007 commentedon Nov 11, 2026
thankss !!
Wizard52007 commentedon Nov 11, 2026
Hi!
While investigating and working on this issue, I discovered that PR #3045 already addresses the underlying problem with a more comprehensive event-driven approach.
I initially implemented a fix independently, but after comparing both implementations, I found that PR #3045 also handles Secret creation/deletion events, avoids caching the full Secret payload, and handles transient API errors.
I also identified an opportunity to strengthen its test coverage. I've added a test that uses the real condition updater and verifies that the
NotReadystatus and associated conditions are actually persisted.I've reached out to the author of PR #3045 to offer this additional test.
Would you recommend contributing the test to the existing PR and allowing it to resolve this issue, or is there any additional work you'd like me to take up independently?
Thanks!