Repository navigation
Prevent upgrade controller from reading ClusterPolicy controller globals - #3129
aryangorwade wants to merge 2 commits into
Conversation
Walkthrough
Priority: Severity of issue fixed: Medium Merge Risk:
|
There was a problem hiding this comment.
Actionable comments posted: 1

Review info
-
Run configuration-
- Source mode: Auto
- Repository: NVIDIA/gpu-operator
- Branch: main
- File: .coderabbit.yaml
- Commit: ff1e017
- Layers: Repository YAML
- Review profile: QUIET
- Plan: Enterprise
- Run ID:
7068ff62-5227-477c-82e1-56da35110ab8
Files selected for processing: 3-
controllers/transforms_test.gocontrollers/upgrade_controller.gocontrollers/upgrade_controller_test.go
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
-
7090683 to
ef9132a
Compare
| driverLabelKey := DriverLabelKey | ||
| driverLabelValue := DriverLabelValue | ||
|
|
||
| if clusterPolicyCtrl.openshift != "" && clusterPolicyCtrl.ocpDriverToolkit.enabled { |
There was a problem hiding this comment.
Does not have be addressed in this PR, but I think with this change clusterPolicyCtrl does not need to be a Global variable, we can probably make it a part of ClusterPolicyController
…controller state Signed-off-by: Aryan Gorwade <gorwadearyan@gmail.com>
ef9132a to
a757d96
Compare
Signed-off-by: Aryan Gorwade <gorwadearyan@gmail.com>
Description
Fixes #3061. Depends on: NVIDIA/k8s-operator-libs#166.
Note: Temporarily pinned to my fork pending k8s-operator-libs#166; will switch to upstream before merging.
There was a chance the upgrade controller could read the ClusterPolicy controller's state while it was actively being edited. That caused a data race, and on OpenShift with DTK it could search by the wrong label at startup. At startup it could also read that state before it was set, so on OpenShift with DTK it searched by the wrong label. It now always uses app.kubernetes.io/component=nvidia-driver, which every driver pod has, so it no longer reads that state.
NVIDIADriver-owned DaemonSets are filtered out by passing inisOwnedByNVIDIADriveras aBuildStateownership filter introduced ink8s-operator-libs#166.Note: This fix was tried before, in bd806dd, and reverted in 96a20af before v23.9.0 shipped. The component label
app.kubernetes.io/component=nvidia-driverwas added in the same release cycle, so every cluster upgrading to v23.9.0 had driver pods created by an older operator, and none of them had the label. Driver pods are OnDelete, so they only get new labels when they're deleted and recreated. With no labeled pods, BuildState couldn't find them and upgrades would have stalled, which is why it was reverted. That's no longer a concern because every driver pod created since v23.9.0 (Oct 2023) has the label, and auto-upgrade has since replaced the older pods.Checklist
make lint)make validate-generated-assets)make validate-modules)Testing
Unit tests added that
app.kubernetes.io/component=nvidia-driver, whateverclusterPolicyCtrlholds: before the first ClusterPolicy reconcile, on OpenShift with DTK, and on OpenShift without DTK.applabel but keeps the component label on both the DaemonSet and its pod template, which the new selector depends on.