Skip to content

Prevent upgrade controller from reading ClusterPolicy controller globals - #3129

Open
aryangorwade wants to merge 2 commits into
NVIDIA:mainfrom
aryangorwade:upgrade-controller-cluster-policy-globals-sync
Open

aryangorwade wants to merge 2 commits into
NVIDIA:mainfrom
aryangorwade:upgrade-controller-cluster-policy-globals-sync

Conversation

@aryangorwade

@aryangorwade aryangorwade commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fixes #3061. Depends on: NVIDIA/k8s-operator-libs#166.
Note: Temporarily pinned to my fork pending k8s-operator-libs#166; will switch to upstream before merging.

There was a chance the upgrade controller could read the ClusterPolicy controller's state while it was actively being edited. That caused a data race, and on OpenShift with DTK it could search by the wrong label at startup. At startup it could also read that state before it was set, so on OpenShift with DTK it searched by the wrong label. It now always uses app.kubernetes.io/component=nvidia-driver, which every driver pod has, so it no longer reads that state. NVIDIADriver-owned DaemonSets are filtered out by passing in isOwnedByNVIDIADriver as a BuildState ownership filter introduced in k8s-operator-libs #166.

Note: This fix was tried before, in bd806dd, and reverted in 96a20af before v23.9.0 shipped. The component label app.kubernetes.io/component=nvidia-driver was added in the same release cycle, so every cluster upgrading to v23.9.0 had driver pods created by an older operator, and none of them had the label. Driver pods are OnDelete, so they only get new labels when they're deleted and recreated. With no labeled pods, BuildState couldn't find them and upgrades would have stalled, which is why it was reverted. That's no longer a concern because every driver pod created since v23.9.0 (Oct 2023) has the label, and auto-upgrade has since replaced the older pods.

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Lint checks passing (make lint)
  • Generated assets in-sync (make validate-generated-assets)
  • Go mod artifacts in-sync (make validate-modules)
  • Test cases are added for new code paths

Testing

Unit tests added that

  • Check that the ClusterPolicy upgrade path always selects driver pods by app.kubernetes.io/component=nvidia-driver, whatever clusterPolicyCtrl holds: before the first ClusterPolicy reconcile, on OpenShift with DTK, and on OpenShift without DTK.
  • Check that the OpenShift Driver Toolkit transform renames the app label but keeps the component label on both the DaemonSet and its pod template, which the new selector depends on.
  • Check that the ClusterPolicy upgrade path drops nodes whose driver DaemonSet is owned by an NVIDIADriver, keeping ClusterPolicy-owned and orphaned driver pods, so leftover NVIDIADriver pods aren't upgraded by ClusterPolicy.

Devin Review

@aryangorwade
aryangorwade requested a review from a team as a code owner October 28, 2026 01:54

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Devin Review

Comment thread controllers/upgrade_controller.go Resolved
Comment thread controllers/upgrade_controller_test.go Resolved
@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Walkthrough

ClusterPolicy-managed driver upgrades now build upgrade state using the driver component label selector for all configurations. The controller no longer selects by the app label or the OpenShift DTK identification label. Tests cover the selector before the first ClusterPolicy reconcile and with DTK enabled or disabled. A transformation test checks that the component label remains set when the pod-template driver label changes.

Priority: Normal

Severity of issue fixed: Medium

Merge Risk: Moderate · up to 70906

Clusters that still run driver pods created before v23.9.0 and upgrade the operator straight to this version could see driver upgrades rejected on every reconcile. Most clusters are unaffected, since pods created since v23.9.0 carry the label. Before merging, confirm that pre-v23.9.0 pods are unsupported, or add a migration path for unlabeled pods.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


Review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between ff1e017 and 7090683.

Files selected for processing: 3
  • controllers/transforms_test.go
  • controllers/upgrade_controller.go
  • controllers/upgrade_controller_test.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Comment thread controllers/upgrade_controller.go Resolved
@aryangorwade
aryangorwade force-pushed the upgrade-controller-cluster-policy-globals-sync branch from 7090683 to ef9132a Compare October 28, 2026 18:15
driverLabelKey := DriverLabelKey
driverLabelValue := DriverLabelValue

if clusterPolicyCtrl.openshift != "" && clusterPolicyCtrl.ocpDriverToolkit.enabled {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does not have be addressed in this PR, but I think with this change clusterPolicyCtrl does not need to be a Global variable, we can probably make it a part of ClusterPolicyController

Comment thread controllers/upgrade_controller_test.go Resolved
Comment thread controllers/upgrade_controller.go Resolved
…controller state

Signed-off-by: Aryan Gorwade <gorwadearyan@gmail.com>
@aryangorwade
aryangorwade force-pushed the upgrade-controller-cluster-policy-globals-sync branch from ef9132a to a757d96 Compare October 28, 2026 21:00
Comment thread controllers/upgrade_controller.go Resolved
Signed-off-by: Aryan Gorwade <gorwadearyan@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Upgrade controller reads ClusterPolicy controller globals without synchronization

2 participants