Skip to content

Reduce operator log volume from per-node and per-Pod lines - #3264

Open
aryangorwade wants to merge 3 commits into
NVIDIA:mainfrom
aryangorwade:quiet-operator-logs
Open

aryangorwade wants to merge 3 commits into
NVIDIA:mainfrom
aryangorwade:quiet-operator-logs

Conversation

@aryangorwade

@aryangorwade aryangorwade commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Addresses #722 with k8s-operator-libs #167. Stacked on top of #3228.

Merge #3228 and #167 first. Then update this PR's vendored k8s-operator-libs dependency to the latest upstream version, and merge.

This PR reduces operator log volume by:

  • Bumping k8s-operator-libs to the logging changes in k8s-operator-libs's #167
  • Moving the NVIDIADriver Node-predicate message, Node labels have been changed, to consts.LogLevelDebug: commit 25f20e4
  • Changing severity of object update logs to debug (eg. "Identical, skipping update"): commit c5c567c

Benchmarks:

NOTE: Removing no-op writes and reducing logging overhead can allow more reconciliation passes during an upgrade because fewer events are combined. Because of this the remaining info-level messages can appear more often, increasing total upgrade log lines despite the improvements.

Node count Scenario Idle lines/s Upgrade lines/s Upgrade seconds Upgrade lines
500 GPU Operator main BEFORE 2026-10-20 (vendoring library older than 42ee2fff224f) 2.300 457.488 232.332 106,289
500 Current GPU Operator main with library bump to 42ee2fff224f 1.706 449.583 233.156 104,823
500 GPU Operator PR #3228 + library PR #167 + this PR 0.147 614.403 227.730 139,918
2,500 GPU Operator main BEFORE 2026-10-20 (vendoring library older than 42ee2fff224f) 4.267 471.600 1,231.013 580,546
2,500 Current GPU Operator main with library bump to 42ee2fff224f 3.348 427.501 1,391.251 594,761
2,500 GPU Operator PR #3228 + library PR #167 + this PR 0.118 574.141 1,415.202 812,525

Measurements used a kind control plane with KWOK workers (node count), two driver DaemonSets, a CPU-only simulated driver-upgrade workflow, and info-level logging with default production Zap sampling. Each selected idle window contained two complete upgrade-library passes.


Devin Review

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@aryangorwade aryangorwade changed the title quiet operator logs Reduce operator log volume from per-node and per-Pod lines Nov 18, 2026
@aryangorwade
aryangorwade force-pushed the quiet-operator-logs branch 2 times, most recently from 2d68ef7 to 205f32d Compare November 18, 2026 01:08
…GPUCluster paths

Signed-off-by: Aryan Gorwade <gorwadearyan@gmail.com>
Signed-off-by: Aryan Gorwade <gorwadearyan@gmail.com>
Signed-off-by: Aryan Gorwade <gorwadearyan@gmail.com>
@aryangorwade
aryangorwade marked this pull request as ready for review November 18, 2026 01:29
@aryangorwade
aryangorwade requested a review from a team as a code owner November 18, 2026 01:29
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

The change adds shared object synchronization and content comparison, then uses these paths in state and controller reconciliation. Existing resources can avoid updates when their content is unchanged. Resource-specific behavior includes create-only ServiceAccounts and preservation of Service ClusterIP. Tests cover repeated syncs, disabled-resource cleanup, typed and unstructured reads, and stale-read handling. The node-label change message now uses debug-level logging.

Merge Risk: Low · up to c5c56

This change avoids no-op writes and cuts log volume. A resource update can briefly be reported as ready based on a slightly stale cached copy before the next reconcile corrects it. Merging is reasonably safe, but address the readiness read, or accept the transient inaccuracy knowingly.

Security architecture risk: Moderate · up to c5c56

Cleanup can now succeed based on a cached “not found” result without requesting deletion. During cache lag, disabling a component or its RBAC can therefore appear complete while permissions remain until another reconciliation. Existing authorization gates and ownership assignments are otherwise preserved in the inspected paths.

Retained concerns

  • Medium · security · inferred: Cached absence can prematurely complete privileged-resource cleanup. If an earlier creation has reached the API server but not the informer cache, the new disabled-state path returns success without issuing Delete. ClusterRole and ClusterRoleBinding callers then report Disabled, and an otherwise-ready reconciliation need not retry. Operator-managed permissions can remain after disablement until another reconciliation occurs. The base issued deletion directly. Later reconciliations and owner garbage collection provide recovery, but do not guarantee timely revocation while the policy remains present.
Security review details

Security Blast Radius

  • inferred — The cleanup concern reaches operator-managed namespaced roles and cluster-scoped roles and bindings. Its security impact is retention of an affected grant after disablement, potentially beyond the operator namespace for cluster-scoped RBAC. The inspected change does not establish a new unauthenticated entrypoint or unrestricted cluster takeover.

Security Findings and Attack Paths

  • inferred — A supported failure sequence is creation of managed RBAC, disablement before that creation becomes cache-visible, and a cached NotFound causing deletion to be skipped. A principal still holding the affected bound identity could retain access during that interval. This is an introduced revocation failure condition, not evidence of exploitation or a newly available low-privilege way to modify policy.

Trust Boundaries and Controls

  • observed — The inspected RBAC callers preserve their component-state and RBAC gates, subject namespace substitutions, and ClusterPolicy controller owner. Managing existing same-named objects without checking their current controller owner predates this PR and is not retained as an introduced concern.

Resilience and Maintainability Implications

  • inferred — Actual read or write errors propagate as NotReady, and a later reconciliation can retry missed cleanup. However, cached NotFound is treated as success, and the inspected ClusterPolicy watches cover primary configuration, Nodes, and owned DaemonSets rather than RBAC changes. Cache catch-up alone therefore does not establish prompt cleanup recovery.

Hardening Proposals

  • proposed — Keep disabled-resource deletion authoritative by issuing deletion directly and tolerating API-server NotFound, or require authoritative absence or an explicit retry before reporting cleanup complete. Apply the same completion principle when a missing read is contradicted by AlreadyExists.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 1

internal/state/state_skel.go: 1

253-254: Minor · Functional Correctness · Quick win

Do not use an older cached generation to report readiness.

If a ready DaemonSet is cached at generation 4, updateObj can write generation 5 before the cache advances. syncObjects then calls getSyncState, which reads generation 4 through this branch. isDaemonSetReady accepts its old status and returns SyncStateReady before generation 5 is observed. Deployment readiness has the same failure. Previously, these unstructured reads bypassed the cache. The production client uses the manager cache, and controller-runtime does not enable read-your-writes consistency by default. (raw.githubusercontent.com)

Keep cached reconciliation reads, but use an uncached readiness read or require the generation returned by the successful update. Add a delayed-cache regression test.

Nitpick comments: 1

controllers/object_controls_test.go: 1

2663-2663: Trivial · Maintainability & Code Quality · Quick win

Count Patch requests in the no-write assertion.

The interceptor configuration counts only Create, Update and Delete. An unnecessary Patch on the second call would leave writes at zero and pass this assertion. Add a Patch interceptor that increments writes and forwards the request.

As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”

Source: Path instructions

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 246a9ca and c5c567c.

Files ignored due to path filters: 12

  • go.sum is excluded by !**/*.sum, !go.sum
  • vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/common_manager.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/consts.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/drain_manager.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/node_upgrade_state_provider.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/pod_manager.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/safe_driver_load_manager.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/upgrade_inplace.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/upgrade_requestor.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/upgrade_state.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/k8s-operator-libs/pkg/upgrade/util.go is excluded by !vendor/**
  • vendor/modules.txt is excluded by !vendor/**
Files selected for processing: 12

  • controllers/nvidiadriver_controller.go
  • controllers/object_controls.go
  • controllers/object_controls_test.go
  • go.mod
  • internal/state/dra_driver_test.go
  • internal/state/driver_manifest_test.go
  • internal/state/driver_sync_test.go
  • internal/state/state_skel.go
  • internal/state/state_skel_reconcile_test.go
  • internal/state/state_skel_test.go
  • internal/utils/sync_object.go
  • internal/utils/sync_object_test.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@aryangorwade aryangorwade self-assigned this Nov 18, 2026
@aryangorwade
aryangorwade marked this pull request as draft November 18, 2026 01:38
@aryangorwade
aryangorwade marked this pull request as ready for review November 18, 2026 05:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant