You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gpu-config [13/13]: deploy the GPU configuration manager from GPUCluster - #3259
Part of #3246 (13/13). Depends on: #3247 to #3258. Review only the last commit.
Adds spec.gpuConfigManager to GPUCluster, the DRA-stack manager
DaemonSet, and the gpu-config init container in the DRA kubelet plugin.
Kubelet plugin pods roll once on upgrade.
Adds a GPU configuration manager for container and DRA stacks. The change defines configuration documents and GPU matching, applies power, clock, and ECC settings through NVML, and publishes node status labels. It adds a validator gate, Kubernetes clients and ConfigMap watches, state persistence, and operator APIs, templates, RBAC, and DaemonSets. The manager is disabled by default in the Helm values. The build includes the manager executable and enables CGO for that command.
Merge Risk: Moderate · up to 25ec5
This change adds a node-level GPU settings manager and an init gate that holds GPU allocators until settings are decided for each driver load. Several decision inputs can be stale or misread: cached DaemonSet and label reads, pod lists bounded by an old node version, and failed init containers counted as success. As a result, GPU settings can be silently skipped for a driver load, or applied while workloads are starting. A malformed env entry can also block the DRA plugin rollout. The feature is off by default, but these issues should be fixed before it is relied on.
Security architecture risk: Minimal · up to 25ec5
The inspected configuration and startup paths preserve driver-load identity, serialize configuration with allocator release, and hold startup after incomplete application. No introduced security bypass or material recovery failure was established. The privileged permissions and non-automatic rollback behavior remain important operational constraints.
Retained concerns
No architecture-level concerns identified.
Security review details
Security Blast Radius
inferred — A compromised manager service-account credential could exercise cluster-wide node patch, pod listing, claim reads, and ResourceSlice listing; scheduling and NODE_NAME are not authorization restrictions. This is a new credential holder, but cluster-wide node mutation already exists in the base DRA plugin and MIG manager. The inspected normal publisher writes only three GPU-configuration status labels on its configured node.
Security Findings and Attack Paths
inferred — Write access to a selected ConfigMap becomes authority over supported GPU settings on the nodes selecting it, potentially an entire enabled fleet for the cluster document. Invalid configuration can hold allocator startup. This follows the exposed configuration contract; no inspected path converts document content into commands, Kubernetes credentials, or an arbitrary node-patch target.
Trust Boundaries and Controls
observed — Document watches are restricted in code to selected names in one namespace and must initially synchronize. Parsing rejects unsupported schema and unknown settings; assignment overlap and supported device ranges are checked before writes. Configuration data does not choose the publisher's node identity.
Resilience and Maintainability Implications
observed — The protocol checks decisions again under the shared lock, rechecks driver identity after opening device handles, and checks identity again before gate release. Prior gate passes and allocator or GPU-pod evidence can force a no-write skip. Gate API-read failures treat the manager as expected rather than bypassing the wait; intentional no-manager and bounded no-module exits record their passes.
Hardening Proposals
proposed — Consider mediating status publication or enforcing a dedicated admission policy so manager credentials cannot modify arbitrary node fields. Treat selected ConfigMap write access as hardware-administration authority. These are defense-in-depth proposals, not verified bypass findings.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 4
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
Caution
Some comments are outside the diff and can’t be posted inline due to GitHub limitations.
⚠️ Outside diff range comments (1)
Minor · Removing the output directory can break the GPU configuration lock. · main.go:530
cmd/nvidia-validator/main.go:530 Minor · Stability & Availability · Quick win
Removing the output directory can break the GPU configuration lock.
--cleanup-all calls os.RemoveAll(outputDirFlag) before any lock is taken. The GPU configuration store locks .gpu-config.lock by pathname and holds flock on the opened file. If cleanup runs while an applier or gate holds that lock, the holder keeps the unlinked file. A later lock attempt creates a new file at the same path and acquires it. Both processes can then hold the exclusive lock and write .gpu-config-state.
Nothing in the repository prevents an operator from running --cleanup-all (or setting CLEANUP_ALL) on a node while GPU configuration is in progress. The cleanup path must not remove the lock or state files while a holder can use them. Acquire the state lock before removal, or exclude the lock and state files from removal.
Other comments: 10
api/nvidia/v1/clusterpolicy_types.go: 1
1833-1834: Minor · Functional Correctness · Quick win
Document when the manager reads the pool label.
If a node moves from training to inference after the manager starts but before its first decision, the manager still uses the training ConfigMap. It sets up the watches from the startup label. The later Node read does not change those watches. State that the manager reads the pool label at startup and that relabeling does not change the selected pool until the manager restarts. (raw.githubusercontent.com)
internal/gpuconfig/config/document.go: 2
177-179: Minor · Data Integrity & Integration · Quick win
Reject entry names that lose their identity during JSON conversion.
YAML v2 decodes a !!binary key into a Go string, so the string-type check accepts it. If that key contains an invalid UTF-8 byte and another entry uses U+FFFD, JSON conversion gives both entries the same key. One entry disappears before Resolve can report an overlap. Reject invalid UTF-8 in both explicit and merged entry names before accepting the document. (raw.githubusercontent.com)
126-127: Minor · Functional Correctness · Quick win
Recognize merged schema versions in the fallback.
If <<: {version: v2} supplies the version and a new field contains a null map key, strict conversion fails. The fallback decodes into yamlv2.MapSlice, which drops the merged version. The manager reports a schema error instead of telling the operator that it does not support v2. Use a merge-aware lookup in the fallback and add a regression case for this input. (raw.githubusercontent.com)
internal/gpuconfig/statefile/file.go: 1
210-210: Minor · Data Integrity & Integration · Quick win
Preserve the apply decision when writing a gate pass.
If the file contains "apply":null followed by "Apply":{"module":7,"value":"applied"}, the reader sees an applied decision for load 7. A gate-only write sorts the keys in the opposite order, so null wins. The file still parses, but the next applier poll no longer sees the load as decided. Decode known state keys by exact spelling, or normalize aliases without changing the effective value of unrelated records. (raw.githubusercontent.com)
internal/gpuconfig/config/match_test.go: 1
81-85: Minor · Maintainability & Code Quality · Quick win
Test a match on the second product-name pattern.
Every product-name selector in these fixtures has one pattern. An implementation that checks only the first pattern would pass. Add a case such as productName: ["NVIDIA L40*", "NVIDIA H100*"] against the H100 fixture and assert that the entry's power limit is selected.
As per coding guidelines, “New behavior needs appropriate test coverage.”
Source: Coding guidelines
internal/gpuconfig/kube/documents_test.go: 2
130-134: Minor · Functional Correctness · Quick win
Assert that the edited document becomes visible before deletion.
The test changes only ResourceVersion. It passes if an update triggers onChange but leaves Snapshot() serving the old data. Change config.yaml and wait for Snapshot() to return the new value before sending the delete.
As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”
Source: Path instructions
172-172: Minor · Functional Correctness · Quick win
Test name scoping on the watch path.
This test calls only ListWithContextFunc. It would still pass if WatchFuncWithContext lost its name selector and started receiving other nodes’ ConfigMap events. Exercise the watch path and assert that it excludes an event for other. (raw.githubusercontent.com)
As per coding guidelines, “New behavior needs appropriate test coverage.”
Source: Coding guidelines
internal/gpuconfig/decision/decide.go: 1
358-358: Minor · Stability & Availability · Quick win
Preserve the retryable load-ID error after an NVML open failure.
If NVML initialization fails and the follow-up load-ID read also fails, openFailed discards the ErrTransient error. The applier then exits on the raw NVML error instead of retrying on the next poll. Return the transient error with the NVML error as context. Keep the fatal NVML result when the load-ID read succeeds and confirms that the load did not change. (raw.githubusercontent.com)
internal/gpuconfig/decision/decide_test.go: 1
528-530: Minor · Functional Correctness · Quick win
Test the idle path with the state lock held.
This case passes if Decide acquires the state lock before checking for an existing decision, because nothing holds the lock. Hold the lock independently and verify, with a bounded wait, that Decide returns Idle before the holder releases it.
As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”
Source: Path instructions
manifests/state-dra-driver/0500_daemonset.yaml: 1
150-153: Minor · Functional Correctness · Quick win
Reject or skip spec.gpuConfigManager.env entries with an empty name.
dedupeEnv keeps every entry, including one with an empty name. The DRA DaemonSet template then renders it as - name: with a null name. The Kubernetes API rejects a container env entry that has no name. The DaemonSet apply then fails for that reconcile, and the DRA kubelet plugin update does not roll out.
The duplicate-name case is already handled: dedupeEnv keeps one entry per name.
The fix must ensure that an empty-named entry cannot reach the rendered env list. Either validation rejects it, or the entry is dropped before rendering. The choice between these two is open, so this comment does not prescribe one.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Major · Functional Correctness · Quick win
Require a successful exit from the toolkit wait.
If kubelet kills toolkit-validation while it is still waiting for toolkit-ready, the device plugin has not advanced. On a reload, this check treats the failed termination as allocator evidence and records a skip for the whole load. Require exit code zero for the toolkit init container. Keep the separate check for evidence that a later container started. (raw.githubusercontent.com)
The reason will be displayed to describe this comment to others. Learn more.
Major · Functional Correctness · Quick win
Do not count a failed gpu-config init container as a gate pass.
On a driver reload, a replacement DRA plugin’s gpu-config init container can terminate with a nonzero exit code before the gate passes. This check reports a pass even if no later container ran. The decider records Skipped and will not apply settings again for that load. Require exit code zero here. Keep the separate check for evidence that a later container ran, and add a nonzero-exit regression case. (kubernetes.io)
The reason will be displayed to describe this comment to others. Learn more.
Major · Functional Correctness · Quick win
Do not use unconstrained cached reads to disable the gate.
With ResourceVersion: "0", Kubernetes can return stale data. After the manager DaemonSet is created, DaemonSetExists can still report it absent. A stale "false" deployment label has the same effect. (kubernetes.io)
Gate.readExpected treats either result as no manager expected. The gate can then record a pass before configuration completes. That pass makes the applier skip configuration for the driver load. The lock prevents concurrent writes, but does not correct the stale expectation. Use Most Recent reads before allowing either result to release the gate. (raw.githubusercontent.com)
The reason will be displayed to describe this comment to others. Learn more.
Major · Data Integrity & Integration · Quick win
Use a current pod list for the GPU safety check.
A Node's metadata.resourceVersion identifies its last modification, not the time of a quorum read. If the Node remains at version 100 and a GPU pod starts at version 200, this request can accept a pod-cache snapshot at version 150. NotOlderThan permits that result. (kubernetes.io)
The decision caller uses this list to detect GPU pods before applying settings. An omitted pod or stale pod status can therefore bypass that check. Request Most Recent list semantics by leaving both resource-version fields unset. Update the test to check this freshness requirement. (raw.githubusercontent.com)
henry118
changed the title
gpu-config: deploy the GPU configuration manager from GPUCluster
gpu-config [13/13]: deploy the GPU configuration manager from GPUCluster
Nov 19, 2026
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3246 (13/13). Depends on: #3247 to #3258. Review only the last commit.
Adds
spec.gpuConfigManagerto GPUCluster, the DRA-stack managerDaemonSet, and the
gpu-configinit container in the DRA kubelet plugin.Kubelet plugin pods roll once on upgrade.