You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gpu-config [12/13]: deploy the GPU configuration manager from ClusterPolicy - #3258
Part of #3246 (12/13). Depends on: #3247 to #3257. Review only the last commit.
Adds spec.gpuConfigManager (disabled by default) to ClusterPolicy, CRDs
and Helm, the manager DaemonSet, the gpu-config init container in the
validator, and the node deploy label. The init container is rendered on
every cluster, so validator pods roll once on upgrade.
This change adds a GPU configuration manager and validator gate. The manager watches node-, pool-, and cluster-level ConfigMaps, validates settings, checks driver and workload state, and applies supported GPU settings through NVML. It records decisions and publishes synchronization and activation labels on nodes. Operator API, Helm values, controller transformations, RBAC, and DaemonSet resources add configuration and deployment wiring.
Merge Risk: Low · up to 8fb79
The GPU configuration manager is off by default, so existing clusters only see a one-time validator rollout. When the manager is enabled, a few narrow timing and edge cases could let settings be written while a validation workload is starting. They could also suppress configuration for a driver load, or let a malformed config document slip past validation. These are small, contained fixes that are worth making before relying on the feature, but they do not block merging a disabled-by-default option.
Security architecture risk: Minimal · up to 8fb79
The feature is disabled by default and coordinates configuration writes with allocator startup. The inspected changes did not establish an introduced security, failure-containment, or rollback defect.
Retained concerns
No architecture-level concerns identified.
Security review details
Security Blast Radius
observed — Normal execution targets the configured node, but the service account's authorization is broader: it can get and patch any Node, list Pods and ResourceSlices cluster-wide, and get ResourceClaims across namespaces. Its namespaced role permits ConfigMap list/watch and DaemonSet list without resource-name restrictions.
Trust Boundaries and Controls
observed — Configuration selection crosses from ClusterPolicy and node-label inputs into privileged GPU application. Production document watches are scoped to the configured operator namespace and individual names. Workload exceptions for applier and DRA operator pods require the configured namespace and expected DaemonSet ownership metadata; arbitrary workload labels alone do not satisfy those exceptions.
Resilience and Maintainability Implications
observed — When an applier is expected, the gate waits for a recorded decision. Failed expectation reads are treated conservatively as expected, and state-read or state-write errors prevent passage. Disabled-manager and bounded no-module passage still record evidence that later application checks use to avoid writing after allocators may have been released. Status labels are administrative reports, not gate authorization inputs.
Hardening Proposals
proposed — Consider admission enforcement or a narrower status-publication mechanism that restricts each manager identity to its assigned node and intended status labels. The current application-level targeting does not constrain the reusable credential's cluster-wide Node patch authority.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
The reason will be displayed to describe this comment to others. Learn more.
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
Other comments: 5
internal/gpuconfig/config/document.go: 1
146-146: Minor · Functional Correctness · Quick win
Preserve merged entry keys during validation.
yamlv2.MapSlice drops merged items during decoding. Thus walkKeys does not inspect entry names supplied through <<. This behavior is present in v2.4.4. (raw.githubusercontent.com)
YAML-to-JSON conversion collapses these distinct keys into "1". The parser can accept one entry instead of rejecting the numeric entry name. The overlap check then cannot see both entries. (raw.githubusercontent.com)
Use a key-preserving representation that retains merged mappings. Add this regression case while preserving valid string-key merges.
internal/gpuconfig/decision/decide_test.go: 1
669-673: Minor · Functional Correctness · Quick win
Reject unexpected errors in success cases.
If a success case returns an unexpected error with Outcome: None, unchanged state and closed NVML, this branch passes and returns before checking wantOutcome. For example, the document-apply case can pass without applying its document.
Use this branch only when the case expects an error. Otherwise, require NoError before checking the successful outcome. As per path instructions, flag “assertions that would still pass if the behavior under test were broken.”
Source: Path instructions
internal/gpuconfig/decision/decide.go: 1
307-308: Minor · Functional Correctness · Quick win
Count validation pods that first appear during the re-check.
The second pass skips every pod whose UID was not a candidate at the first list. runWorkload replaces the CUDA validation pod with a new pod, which has a new UID. If the CUDA step reruns during the wait, the new validation pod is skipped. The decision can then write GPU settings while that workload uses the GPU.
Count validation pods on the second pass even when their UID is not a candidate:
for i := range pods {
- if !candidates[string(pods[i].UID)] {+ if !candidates[string(pods[i].UID)] && !d.isValidationPod(&pods[i]) {
continue
}
Add a regression case in which a validation pod first appears in the second list and blocks the apply.
internal/gpuconfig/kube/client.go: 1
145-146: Minor · Functional Correctness · Quick win
Use a most-recent pod list for the GPU safety check.
The Node's resourceVersion marks the Node's last update, not the read time. NotOlderThan with that value only requires the pod cache to be at or beyond that revision. A validation pod created after the Node's last update can therefore be missing from the list. On the reload path, podCheck accepts that list without a recheck, and settings can be written while the validation workload runs. The window is bounded by watch-cache lag, so it is narrow, but the check has no fallback.
Leave both ResourceVersion and ResourceVersionMatch unset in ListNodePods. Keep the node field selector and the terminal-phase filter. Setting ResourceVersionMatch without ResourceVersion is invalid, so both fields must be removed together. This makes the list most-recent and removes the freshness bound. Also update the ListNodePods contract in decide.go so it no longer promises "at resourceVersion or later," and drop the now-unused resourceVersion argument from the interface and callers.
129-129: Minor · Functional Correctness · Quick win
Treat a terminated init container as passed only when it exited with code 0.
allocatorPastWait (line 129) and gatePassed (line 162) return true for any Terminated state. The predicates are documented as "got past its wait" and "gate whose gate passed," so a nonzero exit does not meet either definition. A reload that observes such a container can record Skipped for that driver load. Decide then returns Idle for the same load, and a retry in that load cannot apply the configuration. This needs a sampled terminated window, so it is intermittent.
Require a zero exit code on both checks:
Suggested fix
- if s, ok := st[toolkitContainer]; ok && s.State.Terminated != nil {+ if s, ok := st[toolkitContainer]; ok && s.State.Terminated != nil && s.State.Terminated.ExitCode == 0 {
- if s, ok := st[GateContainer]; ok && s.State.Terminated != nil {+ if s, ok := st[GateContainer]; ok && s.State.Terminated != nil && s.State.Terminated.ExitCode == 0 {
Add regression cases where the toolkit-validation or gpu-config container terminates with a nonzero exit code. The current terminated() helper sets exit code zero, so it cannot exercise this path.
Change the document and check the published labels.
This callback only calls DocumentChanged; it does not change Documents. The test then checks node GET timing, not publication results. A publisher that reads the node but never updates its labels can pass.
Keep the timing assertions. Also change a previously applied document and assert OutOfSync after the jitter expires.
As per path instructions, flag tests where “the test only checks that a mock was called.”
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 0 remain after this review.
henry118
changed the title
gpu-config: deploy the GPU configuration manager from ClusterPolicy
gpu-config [12/13]: deploy the GPU configuration manager from ClusterPolicy
Nov 19, 2026
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3246 (12/13). Depends on: #3247 to #3257. Review only the last commit.
Adds
spec.gpuConfigManager(disabled by default) to ClusterPolicy, CRDsand Helm, the manager DaemonSet, the
gpu-configinit container in thevalidator, and the node deploy label. The init container is rendered on
every cluster, so validator pods roll once on upgrade.