You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Reduce operator log volume from per-node and per-Pod lines - #3264
Moving the NVIDIADriver Node-predicate message, Node labels have been changed, to consts.LogLevelDebug: commit 25f20e4
Changing severity of object update logs to debug (eg. "Identical, skipping update"): commit c5c567c
Benchmarks:
NOTE: Removing no-op writes and reducing logging overhead can allow more reconciliation passes during an upgrade because fewer events are combined. Because of this the remaining info-level messages can appear more often, increasing total upgrade log lines despite the improvements.
Node count
Scenario
Idle lines/s
Upgrade lines/s
Upgrade seconds
Upgrade lines
500
GPU Operator main BEFORE 2026-10-20 (vendoring library older than 42ee2fff224f)
2.300
457.488
232.332
106,289
500
Current GPU Operator main with library bump to 42ee2fff224f
Measurements used a kind control plane with KWOK workers (node count), two driver DaemonSets, a CPU-only simulated driver-upgrade workflow, and info-level logging with default production Zap sampling. Each selected idle window contained two complete upgrade-library passes.
The change adds shared object synchronization and content comparison, then uses these paths in state and controller reconciliation. Existing resources can avoid updates when their content is unchanged. Resource-specific behavior includes create-only ServiceAccounts and preservation of Service ClusterIP. Tests cover repeated syncs, disabled-resource cleanup, typed and unstructured reads, and stale-read handling. The node-label change message now uses debug-level logging.
Merge Risk: Low · up to c5c56
This change avoids no-op writes and cuts log volume. A resource update can briefly be reported as ready based on a slightly stale cached copy before the next reconcile corrects it. Merging is reasonably safe, but address the readiness read, or accept the transient inaccuracy knowingly.
Security architecture risk: Moderate · up to c5c56
Cleanup can now succeed based on a cached “not found” result without requesting deletion. During cache lag, disabling a component or its RBAC can therefore appear complete while permissions remain until another reconciliation. Existing authorization gates and ownership assignments are otherwise preserved in the inspected paths.
Retained concerns
Medium · security · inferred: Cached absence can prematurely complete privileged-resource cleanup. If an earlier creation has reached the API server but not the informer cache, the new disabled-state path returns success without issuing Delete. ClusterRole and ClusterRoleBinding callers then report Disabled, and an otherwise-ready reconciliation need not retry. Operator-managed permissions can remain after disablement until another reconciliation occurs. The base issued deletion directly. Later reconciliations and owner garbage collection provide recovery, but do not guarantee timely revocation while the policy remains present.
Security review details
Security Blast Radius
inferred — The cleanup concern reaches operator-managed namespaced roles and cluster-scoped roles and bindings. Its security impact is retention of an affected grant after disablement, potentially beyond the operator namespace for cluster-scoped RBAC. The inspected change does not establish a new unauthenticated entrypoint or unrestricted cluster takeover.
Security Findings and Attack Paths
inferred — A supported failure sequence is creation of managed RBAC, disablement before that creation becomes cache-visible, and a cached NotFound causing deletion to be skipped. A principal still holding the affected bound identity could retain access during that interval. This is an introduced revocation failure condition, not evidence of exploitation or a newly available low-privilege way to modify policy.
Trust Boundaries and Controls
observed — The inspected RBAC callers preserve their component-state and RBAC gates, subject namespace substitutions, and ClusterPolicy controller owner. Managing existing same-named objects without checking their current controller owner predates this PR and is not retained as an introduced concern.
Resilience and Maintainability Implications
inferred — Actual read or write errors propagate as NotReady, and a later reconciliation can retry missed cleanup. However, cached NotFound is treated as success, and the inspected ClusterPolicy watches cover primary configuration, Nodes, and owned DaemonSets rather than RBAC changes. Cache catch-up alone therefore does not establish prompt cleanup recovery.
Hardening Proposals
proposed — Keep disabled-resource deletion authoritative by issuing deletion directly and tolerating API-server NotFound, or require authoritative absence or an explicit retry before reporting cleanup complete. Apply the same completion principle when a missing read is contradicted by AlreadyExists.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
The reason will be displayed to describe this comment to others. Learn more.
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
Other comments: 1
internal/state/state_skel.go: 1
253-254: Minor · Functional Correctness · Quick win
Do not use an older cached generation to report readiness.
If a ready DaemonSet is cached at generation 4, updateObj can write generation 5 before the cache advances. syncObjects then calls getSyncState, which reads generation 4 through this branch. isDaemonSetReady accepts its old status and returns SyncStateReady before generation 5 is observed. Deployment readiness has the same failure. Previously, these unstructured reads bypassed the cache. The production client uses the manager cache, and controller-runtime does not enable read-your-writes consistency by default. (raw.githubusercontent.com)
Keep cached reconciliation reads, but use an uncached readiness read or require the generation returned by the successful update. Add a delayed-cache regression test.
The interceptor configuration counts only Create, Update and Delete. An unnecessary Patch on the second call would leave writes at zero and pass this assertion. Add a Patch interceptor that increments writes and forwards the request.
As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”
Reviewing files that changed from the base of the PR and between 246a9ca and c5c567c.
Files ignored due to path filters: 12
go.sum is excluded by !**/*.sum, !go.sum
vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/common_manager.go is excluded by !vendor/**
vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/consts.go is excluded by !vendor/**
vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/drain_manager.go is excluded by !vendor/**
vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/node_upgrade_state_provider.go is excluded by !vendor/**
vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/pod_manager.go is excluded by !vendor/**
vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/safe_driver_load_manager.go is excluded by !vendor/**
vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/upgrade_inplace.go is excluded by !vendor/**
vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/upgrade_requestor.go is excluded by !vendor/**
vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/upgrade_state.go is excluded by !vendor/**
vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/util.go is excluded by !vendor/**
vendor/modules.txt is excluded by !vendor/**
Files selected for processing: 12
controllers/nvidiadriver_controller.go
controllers/object_controls.go
controllers/object_controls_test.go
go.mod
internal/state/dra_driver_test.go
internal/state/driver_manifest_test.go
internal/state/driver_sync_test.go
internal/state/state_skel.go
internal/state/state_skel_reconcile_test.go
internal/state/state_skel_test.go
internal/utils/sync_object.go
internal/utils/sync_object_test.go
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #722 with
k8s-operator-libs#167. Stacked on top of #3228.Merge #3228 and #167 first. Then update this PR's vendored
k8s-operator-libsdependency to the latest upstream version, and merge.This PR reduces operator log volume by:
k8s-operator-libsto the logging changes in k8s-operator-libs's #167Node labels have been changed, toconsts.LogLevelDebug: commit 25f20e4Benchmarks:
NOTE: Removing no-op writes and reducing logging overhead can allow more reconciliation passes during an upgrade because fewer events are combined. Because of this the remaining info-level messages can appear more often, increasing total upgrade log lines despite the improvements.
42ee2fff224f)42ee2fff224f42ee2fff224f)42ee2fff224fMeasurements used a kind control plane with KWOK workers (node count), two driver DaemonSets, a CPU-only simulated driver-upgrade workflow, and info-level logging with default production Zap sampling. Each selected idle window contained two complete upgrade-library passes.