Skip to content

[Bug]: ClusterPolicy and NVIDIADriver reconcilers pass unpopulated CR to SetConditionsError on r.Get failure #3190

Open

Description

@AyushSrivastava1818

Motivation
During transient API server disconnections, network latency, or client rate limiting, ClusterPolicyReconciler and NVIDIADriverReconciler attempt to update status conditions on an unpopulated custom resource object where .metadata.name is empty "". This triggers a secondary error (resource name may not be empty), polluting logs and obscuring the root cause.

Check here if you would like to work on this issue

  • [x]

Describe the bug
In controllers/clusterpolicy_controller.go:104-120:

	// Fetch the ClusterPolicy instance
	instance := &gpuv1.ClusterPolicy{}
	if err := r.Get(ctx, req.NamespacedName, instance); err != nil {
		err = fmt.Errorf("failed to get ClusterPolicy object: %w", err)
		logger.Error(err, "unable to fetch ClusterPolicy")
		clusterPolicyCtrl.operatorMetrics.reconciliationStatus.Set(reconciliationStatusClusterPolicyUnavailable)
		if apierrors.IsNotFound(err) {
			return reconcile.Result{}, nil
		}
		// Error reading the object - requeue the request.
		if condErr := r.conditionUpdater.SetConditionsError(ctx, instance, conditions.ReconcileFailed, err.Error()); condErr != nil {
			logger.Error(condErr, "failed to set condition")
		}
		return reconcile.Result{}, err
	}

And similarly in controllers/nvidiadriver_controller.go:90-104:

	// Get the NvidiaDriver instance from this request
	instance := &nvidiav1alpha1.NVIDIADriver{}
	if err := r.Get(ctx, req.NamespacedName, instance); err != nil {
		if apierrors.IsNotFound(err) {
			return reconcile.Result{}, nil
		}
		wrappedErr := fmt.Errorf("error getting NVIDIADriver object: %w", err)
		logger.Error(err, "error getting NVIDIADriver object")
		instance.Status.State = nvidiav1alpha1.NotReady
		if condErr := r.conditionUpdater.SetConditionsError(ctx, instance, conditions.ReconcileFailed, wrappedErr.Error()); condErr != nil {
			logger.Error(condErr, "failed to set condition")
		}
		return reconcile.Result{}, wrappedErr
	}

When r.Get fails with a non-404 error (e.g. context deadline exceeded, etcdserver: leader changed, or API rate limiting), instance is left as an empty struct pointer (&gpuv1.ClusterPolicy{}) where instance.Name == "".
Passing this unpopulated object to r.conditionUpdater.SetConditionsError causes updateConditions (internal/conditions/clusterpolicy.go:63) to execute:

u.client.Get(ctx, types.NamespacedName{Name: cr.Name}, instance)

Because cr.Name == "", controller-runtime immediately returns an error:
failed to get ClusterPolicy instance for status update: resource name may not be empty.

In contrast, controllers/gpucluster_controller.go:91-95 correctly handles this scenario with the following author note:

	// instance was not populated by the failed Get, so there is no object to
	// update status on; just surface the error for requeue.
	logger.Error(err, "error getting GPUCluster object")
	return ctrl.Result{}, fmt.Errorf("error getting GPUCluster object: %w", err)

To Reproduce

  1. Simulate or induce a transient network failure or timeout between the GPU Operator pod and kube-apiserver while reconciling ClusterPolicy or NVIDIADriver.
  2. Inspect the operator logs.
  3. Observe:
    ERROR controllers.ClusterPolicy unable to fetch ClusterPolicy {"error": "failed to get ClusterPolicy object: ..."}
    ERROR controllers.ClusterPolicy failed to set condition {"error": "failed to get ClusterPolicy instance for status update: resource name may not be empty"}
    

Expected behavior
When r.Get(ctx, req.NamespacedName, instance) fails, instance is unpopulated. The controller should not attempt to call r.conditionUpdater.SetConditionsError with an empty resource name. It should log the fetch failure and return the error to trigger a backoff requeue, consistent with gpucluster_controller.go.

Environment (please provide the following information):

  • GPU Operator Version: v25.3.x / master
  • Kubernetes Distro and Version: K8s v1.31 / v1.32

Information to attach

  • Operator error logs showing resource name may not be empty

Proposed Fix:

In controllers/clusterpolicy_controller.go:

 		if apierrors.IsNotFound(err) {
 			// Request object not found, could have been deleted after reconcile request.
 			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
 			// Return and don't requeue
 			return reconcile.Result{}, nil
 		}
-		// Error reading the object - requeue the request.
-		if condErr := r.conditionUpdater.SetConditionsError(ctx, instance, conditions.ReconcileFailed, err.Error()); condErr != nil {
-			logger.Error(condErr, "failed to set condition")
-		}
+		// instance was not populated by the failed Get, so there is no object to
+		// update status on; just surface the error for requeue.
 		return reconcile.Result{}, err

In controllers/nvidiadriver_controller.go:

 		if apierrors.IsNotFound(err) {
 			// Request object not found, could have been deleted after reconcile request.
 			return reconcile.Result{}, nil
 		}
 		wrappedErr := fmt.Errorf("error getting NVIDIADriver object: %w", err)
 		logger.Error(err, "error getting NVIDIADriver object")
-		instance.Status.State = nvidiav1alpha1.NotReady
-		if condErr := r.conditionUpdater.SetConditionsError(ctx, instance, conditions.ReconcileFailed, wrappedErr.Error()); condErr != nil {
-			logger.Error(condErr, "failed to set condition")
-		}
+		// instance was not populated by the failed Get, so there is no object to
+		// update status on; just surface the error for requeue.
 		return reconcile.Result{}, wrappedErr

Activity

  1. added
    bugIssue/PR to expose/discuss/fix a bug
    needs-triageissue or PR has not been assigned a priority-px label
    on Nov 8, 2026
  2. AyushSrivastava1818 commented on Nov 8, 2026

    @AyushSrivastava1818
    Author

    /assign

  3. added
    acceptedThe issue is approved for work. Contributors can claim it.
    and removed
    needs-triageissue or PR has not been assigned a priority-px label
    on Nov 16, 2026
  4. kvalliyurnatt commented on Nov 16, 2026

    @kvalliyurnatt
    Contributor

    @AyushSrivastava1818 assigning to you, let me know if you are still interested in working on this. I am assigning this a low priority as this adds one log line and even that is rare, this does not cause functional issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

acceptedThe issue is approved for work. Contributors can claim it.bugIssue/PR to expose/discuss/fix a bug

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions