You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
gpu-config [7/13]: add the gate that holds GPU allocators - #3253
Part of #3246 (7/13). Depends on: #3247. Review only the last commit.
Adds internal/gpuconfig/gate: the logic of the gpu-config init
container, which holds GPU allocators until the manager has decided for the
current driver load. Not used yet.
The change adds a GPU configuration lifecycle. It parses and validates configuration documents, resolves settings for GPUs, and applies supported power, clock, and ECC settings through NVML. It adds driver-load and readiness checks, locked state-file persistence, allocator gating, and decision logic based on Kubernetes evidence and GPU pods. Tests cover parsing, setting application, state transitions, gate behavior, and decision outcomes.
Merge Risk: Low · up to 0f881
This adds GPU configuration parsing, NVML setting application, and allocator gating that are not yet wired into the operator, so there is no immediate production impact. Before this logic is enabled, fix three issues. A failed gate or toolkit-validation container is currently treated as successful completion. Merged YAML entries can bypass the entry-name validation. One decision test can pass despite an unexpected error. The change is safe to merge with these follow-ups tracked.
Security architecture risk: Minimal · up to 0f881
The inspected change does not establish an introduced security weakness or broken coordination guarantee. Configuration and allocator release share an exclusive lock, and release is recorded before proceeding. Production activation is deferred; configuration permissions and deployment integration still require assessment when connected.
Retained concerns
No architecture-level concerns identified.
Security review details
Security Blast Radius
inferred — Once integrated with sufficient NVML authority, one decision can affect settings across all GPUs enumerated by its local NVML session, including pending ECC mode. The source does not establish tenant-scoped authorization or cluster-wide document distribution, so neither tenant isolation nor a cluster-wide attacker reach is verified.
Trust Boundaries and Controls
observed — The shared state mediates the allocator-to-configuration boundary. Gates record a pass before returning; decisions reject current-load passes and passes made without a decision. Earlier-load releases trigger additional allocator and workload checks. Failed gate API reads treat the manager as expected rather than authorizing immediate release.
observed — The document enters through an injected snapshot callback. Its ConfigMap read path and writer authorization remain unresolved; parsing is validation, not proof of source authority. No concrete attacker-to-document-to-GPU path is established by this implementation alone.
Resilience and Maintainability Implications
observed — Release without a decision is an explicit policy when no manager is expected or the no-module bound expires. It still requires a successfully persisted pass, which later decisions consume as exclusion evidence. Driver-load rechecks and conservative leftover pass records contain reload races; gates release their locks before polling or sleeping.
Hardening Proposals
proposed — At production integration, explicitly constrain ConfigMap writers and shared-state directory writers, verify that every participant uses the same lock and boot-scoped storage, and define operator recovery for partial GPU application before enabling allocator gating. These are integration proposals, not observed vulnerabilities.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 2
Note
Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.
Other comments: 1
internal/gpuconfig/decision/decide_test.go: 1
669-673: Minor · Functional Correctness · Quick win
Reject unexpected errors in success cases.
If a success case returns None and an error before writing state, this branch can pass and return. It never checks the expected outcome or persisted configuration.
When neither wantFatal nor wantErrIs expects an error, call require.NoError before this branch.
As per path instructions, flag “assertions that would still pass if the behavior under test were broken”.
A DeviceByIndex(0) implementation that returns nil, nil would pass this assertion. Keep the returned device and check that UUID() returns the identity configured on dev. This verifies that callers receive a usable wrapper for the selected device.
As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”
Reviewing files that changed from the base of the PR and between 246a9ca and 0f88146.
Files ignored due to path filters: 39
go.sum is excluded by !**/*.sum, !go.sum
vendor/github.com/NVIDIA/go-nvml/LICENSE is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_linux.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_other.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/api.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers.h is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers_static.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const_static.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/device.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/doc.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/dynamicLibrary_mock.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/event_set.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/gpm.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/init.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/lib.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/memory_limits.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/README.md is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/computeinstance.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/device.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/eventset.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/extendedinterface.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpmsample.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpuinstance.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/interface.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/unit.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgpuinstance.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgputypeid.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.h is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/refcount.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/return.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/system.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/types_gen.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/unit.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/vgpu.go is excluded by !vendor/**
vendor/github.com/NVIDIA/go-nvml/pkg/nvml/zz_generated.api.go is excluded by !vendor/**, !**/zz_generated.*.go
vendor/modules.txt is excluded by !vendor/**
Files selected for processing: 28
go.mod
internal/gpuconfig/apply/apply.go
internal/gpuconfig/apply/apply_test.go
internal/gpuconfig/apply/fake/fake.go
internal/gpuconfig/apply/nvml.go
internal/gpuconfig/apply/nvmllib/nvmllib.go
internal/gpuconfig/apply/nvmllib/nvmllib_test.go
internal/gpuconfig/config/document.go
internal/gpuconfig/config/document_test.go
internal/gpuconfig/config/match.go
internal/gpuconfig/config/match_test.go
internal/gpuconfig/config/schema.go
internal/gpuconfig/decision/decide.go
internal/gpuconfig/decision/decide_test.go
internal/gpuconfig/decision/evidence.go
internal/gpuconfig/decision/helpers_test.go
internal/gpuconfig/decision/pods.go
internal/gpuconfig/gate/gate.go
internal/gpuconfig/gate/gate_test.go
internal/gpuconfig/loadid/driver.go
internal/gpuconfig/loadid/loadid.go
internal/gpuconfig/loadid/loadid_test.go
internal/gpuconfig/statefile/file.go
internal/gpuconfig/statefile/file_test.go
internal/gpuconfig/statefile/state.go
internal/gpuconfig/statefile/state_test.go
internal/gpuconfig/status/sync.go
internal/gpuconfig/status/sync_test.go
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 5 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Major · Data Integrity & Integration · Quick win
Validate merged entry names before JSON key conversion.
In YAML v2.4.4, MapSlice drops merged items when mappingSlice replaces its output slice. Thus walkKeys cannot inspect entry names supplied through a merge. (raw.githubusercontent.com)
The strict conversion accepts the distinct YAML keys, then collapses both to JSON key "1". This bypasses the non-string entry-name check. Resolve sees one entry, so it cannot detect the conflicting settings. If both limits are supported, the surviving limit can be applied. (raw.githubusercontent.com)
Walk a representation that retains merged keys and their original types. Reject this document while preserving valid string-key merges. Add this regression case.
The reason will be displayed to describe this comment to others. Learn more.
Major · Functional Correctness · Quick win
Require successful termination of the wait container.
On a reload, a gpu-config container that exits with ExitCode=1 satisfies this check even if all later containers still wait. evidence then returns a skip reason. A failed toolkit-validation container causes the same false-positive at Line 129. Termination alone does not establish successful init-container completion. (kubernetes.io)
Require ExitCode == 0 for these two completion checks. Preserve the separate evidence from later containers. Add regression cases for failed waits.
henry118
changed the title
gpu-config: add the gate that holds GPU allocators
gpu-config [7/13]: add the gate that holds GPU allocators
Nov 19, 2026
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3246 (7/13). Depends on: #3247. Review only the last commit.
Adds
internal/gpuconfig/gate: the logic of thegpu-configinitcontainer, which holds GPU allocators until the manager has decided for the
current driver load. Not used yet.