Skip to content

Read driver upgrade state from the manager cache - #3217

Draft
rajathagasthya wants to merge 1 commit into
NVIDIA:mainfrom
rajathagasthya:upgrade-shared-cache-client
Draft

rajathagasthya wants to merge 1 commit into
NVIDIA:mainfrom
rajathagasthya:upgrade-shared-cache-client

Conversation

@rajathagasthya

@rajathagasthya rajathagasthya commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Issue

None.

Description

The upgrade state manager reads the driver DaemonSets, the driver Pods, and every node with a driver pod from the API server on every pass, also when no upgrade is in progress. Pass the manager's client to NewClusterUpgradeStateManager so that it reads these objects from the cache. The Node cache transform now keeps spec.unschedulable and the Ready condition, which the upgrade library reads to count unavailable nodes and to uncordon.

Companion to NVIDIA/k8s-operator-libs#164, which adds a client parameter to NewClusterUpgradeStateManager. This PR vendors that change as v0.0.0-20261113163306-991fa4a9a29a and must move to a library release before it merges. The vendor update also brings in the k8s-operator-libs commits merged since the current version.

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Lint checks passing (make lint) — the only findings are 3 staticcheck SA4023 findings in cmd/nvidia-validator/main.go, which upstream main also has
  • Generated assets in-sync (make validate-generated-assets)
  • Go mod artifacts in-sync (make validate-modules)
  • Test cases are added for new code paths

Testing

  • TestCache checks that the Node transform keeps spec.unschedulable and the Ready condition.
  • go test ./controllers/... ./cmd/... passes.
  • Scale test with 200 fake GPU nodes (kwok), one driver upgrade per test window: Node GETs to the API server fell from 10,049 to about 470 per upgrade, and the maxUnavailable limit and uncordon worked as before. The full results are in Let callers pass the client used by the upgrade state manager k8s-operator-libs#164.

Devin Review

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 2 potential issues.

Devin Review

Comment thread cmd/gpu-operator/main.go Resolved
Comment thread cmd/gpu-operator/main.go Outdated
mgr.GetEventRecorderFor("nvidia-gpu-operator"),
upgrade.StateOptions{},
// Read Nodes and the driver DaemonSets and Pods from the manager cache.
upgrade.StateOptions{Client: mgr.GetClient()},

@devin-ai-integration devin-ai-integration Bot Nov 10, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Upgrade cache path remains untested

TestCache checks Node reads, but no test runs BuildState against this cache. The DaemonSet and Pod cache path lacks regression coverage.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration[bot]

This comment was marked as resolved.

@rajathagasthya rajathagasthya added the performance Issues/PRs that are related to performance label Nov 10, 2026
@rajathagasthya
rajathagasthya force-pushed the upgrade-shared-cache-client branch from 96b31a1 to f1609fc Compare November 13, 2026 16:37
The upgrade state manager reads the driver DaemonSets, the driver Pods,
and every node with a driver pod from the API server on every pass,
also when no upgrade is in progress. Pass the manager's client to
NewClusterUpgradeStateManager so that it reads these objects from the
cache. The Node cache transform now keeps spec.unschedulable and the
Ready condition, which the upgrade library reads to count unavailable
nodes and to uncordon. The vendored k8s-operator-libs is an unreleased
commit and needs a release before this merges.

Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>
@rajathagasthya
rajathagasthya force-pushed the upgrade-shared-cache-client branch from f1609fc to 731047f Compare November 13, 2026 16:41

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Issues/PRs that are related to performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant