Repository navigation
Description
Motivation
During transient API server disconnections, network latency, or client rate limiting, ClusterPolicyReconciler and NVIDIADriverReconciler attempt to update status conditions on an unpopulated custom resource object where .metadata.name is empty "". This triggers a secondary error (resource name may not be empty), polluting logs and obscuring the root cause.
Check here if you would like to work on this issue
- [x]
Describe the bug
In controllers/clusterpolicy_controller.go:104-120:
// Fetch the ClusterPolicy instance
instance := &gpuv1.ClusterPolicy{}
if err := r.Get(ctx, req.NamespacedName, instance); err != nil {
err = fmt.Errorf("failed to get ClusterPolicy object: %w", err)
logger.Error(err, "unable to fetch ClusterPolicy")
clusterPolicyCtrl.operatorMetrics.reconciliationStatus.Set(reconciliationStatusClusterPolicyUnavailable)
if apierrors.IsNotFound(err) {
return reconcile.Result{}, nil
}
// Error reading the object - requeue the request.
if condErr := r.conditionUpdater.SetConditionsError(ctx, instance, conditions.ReconcileFailed, err.Error()); condErr != nil {
logger.Error(condErr, "failed to set condition")
}
return reconcile.Result{}, err
}And similarly in controllers/nvidiadriver_controller.go:90-104:
// Get the NvidiaDriver instance from this request
instance := &nvidiav1alpha1.NVIDIADriver{}
if err := r.Get(ctx, req.NamespacedName, instance); err != nil {
if apierrors.IsNotFound(err) {
return reconcile.Result{}, nil
}
wrappedErr := fmt.Errorf("error getting NVIDIADriver object: %w", err)
logger.Error(err, "error getting NVIDIADriver object")
instance.Status.State = nvidiav1alpha1.NotReady
if condErr := r.conditionUpdater.SetConditionsError(ctx, instance, conditions.ReconcileFailed, wrappedErr.Error()); condErr != nil {
logger.Error(condErr, "failed to set condition")
}
return reconcile.Result{}, wrappedErr
}When r.Get fails with a non-404 error (e.g. context deadline exceeded, etcdserver: leader changed, or API rate limiting), instance is left as an empty struct pointer (&gpuv1.ClusterPolicy{}) where instance.Name == "".
Passing this unpopulated object to r.conditionUpdater.SetConditionsError causes updateConditions (internal/conditions/clusterpolicy.go:63) to execute:
u.client.Get(ctx, types.NamespacedName{Name: cr.Name}, instance)Because cr.Name == "", controller-runtime immediately returns an error:
failed to get ClusterPolicy instance for status update: resource name may not be empty.
In contrast, controllers/gpucluster_controller.go:91-95 correctly handles this scenario with the following author note:
// instance was not populated by the failed Get, so there is no object to
// update status on; just surface the error for requeue.
logger.Error(err, "error getting GPUCluster object")
return ctrl.Result{}, fmt.Errorf("error getting GPUCluster object: %w", err)To Reproduce
- Simulate or induce a transient network failure or timeout between the GPU Operator pod and kube-apiserver while reconciling
ClusterPolicyorNVIDIADriver. - Inspect the operator logs.
- Observe:
ERROR controllers.ClusterPolicy unable to fetch ClusterPolicy {"error": "failed to get ClusterPolicy object: ..."} ERROR controllers.ClusterPolicy failed to set condition {"error": "failed to get ClusterPolicy instance for status update: resource name may not be empty"}
Expected behavior
When r.Get(ctx, req.NamespacedName, instance) fails, instance is unpopulated. The controller should not attempt to call r.conditionUpdater.SetConditionsError with an empty resource name. It should log the fetch failure and return the error to trigger a backoff requeue, consistent with gpucluster_controller.go.
Environment (please provide the following information):
- GPU Operator Version: v25.3.x / master
- Kubernetes Distro and Version: K8s v1.31 / v1.32
Information to attach
- Operator error logs showing
resource name may not be empty
Proposed Fix:
In controllers/clusterpolicy_controller.go:
if apierrors.IsNotFound(err) {
// Request object not found, could have been deleted after reconcile request.
// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
// Return and don't requeue
return reconcile.Result{}, nil
}
- // Error reading the object - requeue the request.
- if condErr := r.conditionUpdater.SetConditionsError(ctx, instance, conditions.ReconcileFailed, err.Error()); condErr != nil {
- logger.Error(condErr, "failed to set condition")
- }
+ // instance was not populated by the failed Get, so there is no object to
+ // update status on; just surface the error for requeue.
return reconcile.Result{}, errIn controllers/nvidiadriver_controller.go:
if apierrors.IsNotFound(err) {
// Request object not found, could have been deleted after reconcile request.
return reconcile.Result{}, nil
}
wrappedErr := fmt.Errorf("error getting NVIDIADriver object: %w", err)
logger.Error(err, "error getting NVIDIADriver object")
- instance.Status.State = nvidiav1alpha1.NotReady
- if condErr := r.conditionUpdater.SetConditionsError(ctx, instance, conditions.ReconcileFailed, wrappedErr.Error()); condErr != nil {
- logger.Error(condErr, "failed to set condition")
- }
+ // instance was not populated by the failed Get, so there is no object to
+ // update status on; just surface the error for requeue.
return reconcile.Result{}, wrappedErr
Activity
AyushSrivastava1818 commentedon Nov 8, 2026
/assign
kvalliyurnatt commentedon Nov 16, 2026
@AyushSrivastava1818 assigning to you, let me know if you are still interested in working on this. I am assigning this a low priority as this adds one log line and even that is rare, this does not cause functional issues.