Skip to content

gpu-config [12/13]: deploy the GPU configuration manager from ClusterPolicy - #3258

Open
henry118 wants to merge 12 commits into
NVIDIA:mainfrom
henry118:gpu-config/12-clusterpolicy
Open

henry118 wants to merge 12 commits into
NVIDIA:mainfrom
henry118:gpu-config/12-clusterpolicy

Conversation

@henry118

@henry118 henry118 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Part of #3246 (12/13). Depends on: #3247 to #3257. Review only the last commit.

Adds spec.gpuConfigManager (disabled by default) to ClusterPolicy, CRDs
and Helm, the manager DaemonSet, the gpu-config init container in the
validator, and the node deploy label. The init container is rendered on
every cluster, so validator pods roll once on upgrade.


Devin Review

@henry118
henry118 requested a review from a team as a code owner November 17, 2026 19:07

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

This change adds a GPU configuration manager and validator gate. The manager watches node-, pool-, and cluster-level ConfigMaps, validates settings, checks driver and workload state, and applies supported GPU settings through NVML. It records decisions and publishes synchronization and activation labels on nodes. Operator API, Helm values, controller transformations, RBAC, and DaemonSet resources add configuration and deployment wiring.

Merge Risk: Low · up to 8fb79

The GPU configuration manager is off by default, so existing clusters only see a one-time validator rollout. When the manager is enabled, a few narrow timing and edge cases could let settings be written while a validation workload is starting. They could also suppress configuration for a driver load, or let a malformed config document slip past validation. These are small, contained fixes that are worth making before relying on the feature, but they do not block merging a disabled-by-default option.

Security architecture risk: Minimal · up to 8fb79

The feature is disabled by default and coordinates configuration writes with allocator startup. The inspected changes did not establish an introduced security, failure-containment, or rollback defect.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • observed — Normal execution targets the configured node, but the service account's authorization is broader: it can get and patch any Node, list Pods and ResourceSlices cluster-wide, and get ResourceClaims across namespaces. Its namespaced role permits ConfigMap list/watch and DaemonSet list without resource-name restrictions.

Trust Boundaries and Controls

  • observed — Configuration selection crosses from ClusterPolicy and node-label inputs into privileged GPU application. Production document watches are scoped to the configured operator namespace and individual names. Workload exceptions for applier and DRA operator pods require the configured namespace and expected DaemonSet ownership metadata; arbitrary workload labels alone do not satisfy those exceptions.

Resilience and Maintainability Implications

  • observed — When an applier is expected, the gate waits for a recorded decision. Failed expectation reads are treated conservatively as expected, and state-read or state-write errors prevent passage. Disabled-manager and bounded no-module passage still record evidence that later application checks use to avoid writing after allocators may have been released. Status labels are administrative reports, not gate authorization inputs.

Hardening Proposals

  • proposed — Consider admission enforcement or a narrower status-publication mechanism that restricts each manager identity to its assigned node and intended status labels. The current application-level targeting does not constrain the reusable credential's cluster-wide Node patch authority.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 5

internal/gpuconfig/config/document.go: 1

146-146: Minor · Functional Correctness · Quick win

Preserve merged entry keys during validation.

yamlv2.MapSlice drops merged items during decoding. Thus walkKeys does not inspect entry names supplied through <<. This behavior is present in v2.4.4. (raw.githubusercontent.com)

For example:

version: v1
gpus:
  <<:
    1: {deviceSelectors: [{productName: ["*"]}], settings: {powerLimit: 250}}
    "1": {deviceSelectors: [{productName: ["*"]}], settings: {powerLimit: 300}}

YAML-to-JSON conversion collapses these distinct keys into "1". The parser can accept one entry instead of rejecting the numeric entry name. The overlap check then cannot see both entries. (raw.githubusercontent.com)

Use a key-preserving representation that retains merged mappings. Add this regression case while preserving valid string-key merges.

internal/gpuconfig/decision/decide_test.go: 1

669-673: Minor · Functional Correctness · Quick win

Reject unexpected errors in success cases.

If a success case returns an unexpected error with Outcome: None, unchanged state and closed NVML, this branch passes and returns before checking wantOutcome. For example, the document-apply case can pass without applying its document.

Use this branch only when the case expects an error. Otherwise, require NoError before checking the successful outcome. As per path instructions, flag “assertions that would still pass if the behavior under test were broken.”

Source: Path instructions

internal/gpuconfig/decision/decide.go: 1

307-308: Minor · Functional Correctness · Quick win

Count validation pods that first appear during the re-check.

The second pass skips every pod whose UID was not a candidate at the first list. runWorkload replaces the CUDA validation pod with a new pod, which has a new UID. If the CUDA step reruns during the wait, the new validation pod is skipped. The decision can then write GPU settings while that workload uses the GPU.

Count validation pods on the second pass even when their UID is not a candidate:

 	for i := range pods {
-		if !candidates[string(pods[i].UID)] {
+		if !candidates[string(pods[i].UID)] && !d.isValidationPod(&pods[i]) {
 			continue
 		}

Add a regression case in which a validation pod first appears in the second list and blocks the apply.

internal/gpuconfig/kube/client.go: 1

145-146: Minor · Functional Correctness · Quick win

Use a most-recent pod list for the GPU safety check.

The Node's resourceVersion marks the Node's last update, not the read time. NotOlderThan with that value only requires the pod cache to be at or beyond that revision. A validation pod created after the Node's last update can therefore be missing from the list. On the reload path, podCheck accepts that list without a recheck, and settings can be written while the validation workload runs. The window is bounded by watch-cache lag, so it is narrow, but the check has no fallback.

Leave both ResourceVersion and ResourceVersionMatch unset in ListNodePods. Keep the node field selector and the terminal-phase filter. Setting ResourceVersionMatch without ResourceVersion is invalid, so both fields must be removed together. This makes the list most-recent and removes the freshness bound. Also update the ListNodePods contract in decide.go so it no longer promises "at resourceVersion or later," and drop the now-unused resourceVersion argument from the interface and callers.

Suggested fix

 	opts := &client.ListOptions{
 		FieldSelector: fields.OneTermEqualSelector(NodeNameField, node),
-		Raw: &metav1.ListOptions{
-			ResourceVersion:      resourceVersion,
-			ResourceVersionMatch: metav1.ResourceVersionMatchNotOlderThan,
-		},
 	}
internal/gpuconfig/decision/pods.go: 1

129-129: Minor · Functional Correctness · Quick win

Treat a terminated init container as passed only when it exited with code 0.

allocatorPastWait (line 129) and gatePassed (line 162) return true for any Terminated state. The predicates are documented as "got past its wait" and "gate whose gate passed," so a nonzero exit does not meet either definition. A reload that observes such a container can record Skipped for that driver load. Decide then returns Idle for the same load, and a retry in that load cannot apply the configuration. This needs a sampled terminated window, so it is intermittent.

Require a zero exit code on both checks:

Suggested fix

-	if s, ok := st[toolkitContainer]; ok && s.State.Terminated != nil {
+	if s, ok := st[toolkitContainer]; ok && s.State.Terminated != nil && s.State.Terminated.ExitCode == 0 {
-		if s, ok := st[GateContainer]; ok && s.State.Terminated != nil {
+		if s, ok := st[GateContainer]; ok && s.State.Terminated != nil && s.State.Terminated.ExitCode == 0 {

Add regression cases where the toolkit-validation or gpu-config container terminates with a nonzero exit code. The current terminated() helper sets exit code zero, so it cannot exercise this path.

Nitpick comments: 1

internal/gpuconfig/applier/applier_test.go: 1

459-459: Trivial · Maintainability & Code Quality · Quick win

Change the document and check the published labels.

This callback only calls DocumentChanged; it does not change Documents. The test then checks node GET timing, not publication results. A publisher that reads the node but never updates its labels can pass.

Keep the timing assertions. Also change a previously applied document and assert OutOfSync after the jitter expires.

As per path instructions, flag tests where “the test only checks that a mock was called.”

Source: Path instructions

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 246a9ca and 8fb79ee.

Files ignored due to path filters: 43

  • api/nvidia/v1/zz_generated.deepcopy.go is excluded by !**/zz_generated.*.go
  • bundle/manifests/nvidia.com_clusterpolicies.yaml is excluded by !bundle/manifests/nvidia.com_*.yaml
  • config/crd/bases/nvidia.com_clusterpolicies.yaml is excluded by !config/crd/bases/**
  • deployments/gpu-operator/crds/nvidia.com_clusterpolicies.yaml is excluded by !deployments/gpu-operator/crds/**
  • go.sum is excluded by !**/*.sum, !go.sum
  • vendor/github.com/NVIDIA/go-nvml/LICENSE is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_linux.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_other.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/api.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers.h is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers_static.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const_static.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/device.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/doc.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/dynamicLibrary_mock.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/event_set.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/gpm.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/init.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/lib.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/memory_limits.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/README.md is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/computeinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/device.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/eventset.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/extendedinterface.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpmsample.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpuinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/interface.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/unit.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgpuinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgputypeid.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.h is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/refcount.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/return.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/system.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/types_gen.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/unit.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/vgpu.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/zz_generated.api.go is excluded by !vendor/**, !**/zz_generated.*.go
  • vendor/modules.txt is excluded by !vendor/**
Files selected for processing: 60

  • Makefile
  • THIRD_PARTY_NOTICES.md
  • api/nvidia/v1/clusterpolicy_types.go
  • assets/state-gpu-config-manager/0100_service_account.yaml
  • assets/state-gpu-config-manager/0200_role.yaml
  • assets/state-gpu-config-manager/0210_clusterrole.yaml
  • assets/state-gpu-config-manager/0300_rolebinding.yaml
  • assets/state-gpu-config-manager/0310_clusterrolebinding.yaml
  • assets/state-gpu-config-manager/0500_daemonset.yaml
  • assets/state-operator-validation/0500_daemonset.yaml
  • bundle/manifests/gpu-operator-certified.clusterserviceversion.yaml
  • cmd/gpu-config-manager/main.go
  • cmd/gpu-config-manager/main_test.go
  • cmd/nvidia-validator/gpuconfig.go
  • cmd/nvidia-validator/gpuconfig_names_test.go
  • cmd/nvidia-validator/gpuconfig_test.go
  • cmd/nvidia-validator/main.go
  • controllers/gpuconfig_test.go
  • controllers/nodelabeling_controller_test.go
  • controllers/object_controls.go
  • controllers/state_manager.go
  • deployments/gpu-operator/templates/clusterpolicy.yaml
  • deployments/gpu-operator/values.yaml
  • docker/Dockerfile
  • go.mod
  • internal/gpuconfig/applier/applier.go
  • internal/gpuconfig/applier/applier_test.go
  • internal/gpuconfig/apply/apply.go
  • internal/gpuconfig/apply/apply_test.go
  • internal/gpuconfig/apply/fake/fake.go
  • internal/gpuconfig/apply/nvml.go
  • internal/gpuconfig/apply/nvmllib/nvmllib.go
  • internal/gpuconfig/apply/nvmllib/nvmllib_test.go
  • internal/gpuconfig/config/document.go
  • internal/gpuconfig/config/document_test.go
  • internal/gpuconfig/config/match.go
  • internal/gpuconfig/config/match_test.go
  • internal/gpuconfig/config/schema.go
  • internal/gpuconfig/decision/decide.go
  • internal/gpuconfig/decision/decide_test.go
  • internal/gpuconfig/decision/evidence.go
  • internal/gpuconfig/decision/helpers_test.go
  • internal/gpuconfig/decision/pods.go
  • internal/gpuconfig/gate/gate.go
  • internal/gpuconfig/gate/gate_test.go
  • internal/gpuconfig/kube/client.go
  • internal/gpuconfig/kube/client_test.go
  • internal/gpuconfig/kube/documents.go
  • internal/gpuconfig/kube/documents_test.go
  • internal/gpuconfig/loadid/driver.go
  • internal/gpuconfig/loadid/loadid.go
  • internal/gpuconfig/loadid/loadid_test.go
  • internal/gpuconfig/publish/publish.go
  • internal/gpuconfig/publish/publish_test.go
  • internal/gpuconfig/statefile/file.go
  • internal/gpuconfig/statefile/file_test.go
  • internal/gpuconfig/statefile/state.go
  • internal/gpuconfig/statefile/state_test.go
  • internal/gpuconfig/status/sync.go
  • internal/gpuconfig/status/sync_test.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 0 remain after this review.

@henry118 henry118 self-assigned this Nov 17, 2026
@henry118
henry118 force-pushed the gpu-config/12-clusterpolicy branch from 8fb79ee to e036a1e Compare November 18, 2026 23:11
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
@henry118
henry118 force-pushed the gpu-config/12-clusterpolicy branch from e036a1e to 375c0f9 Compare November 19, 2026 00:16
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
…r step

Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
@henry118
henry118 force-pushed the gpu-config/12-clusterpolicy branch from 375c0f9 to 1091b51 Compare November 19, 2026 00:50
@henry118 henry118 changed the title gpu-config: deploy the GPU configuration manager from ClusterPolicy gpu-config [12/13]: deploy the GPU configuration manager from ClusterPolicy Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant