Skip to content

gpu-config [7/13]: add the gate that holds GPU allocators - #3253

Open
henry118 wants to merge 7 commits into
NVIDIA:mainfrom
henry118:gpu-config/07-gate
Open

henry118 wants to merge 7 commits into
NVIDIA:mainfrom
henry118:gpu-config/07-gate

Conversation

@henry118

@henry118 henry118 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Part of #3246 (7/13). Depends on: #3247. Review only the last commit.

Adds internal/gpuconfig/gate: the logic of the gpu-config init
container, which holds GPU allocators until the manager has decided for the
current driver load. Not used yet.


Devin Review

@henry118
henry118 requested a review from a team as a code owner November 17, 2026 18:57

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

The change adds a GPU configuration lifecycle. It parses and validates configuration documents, resolves settings for GPUs, and applies supported power, clock, and ECC settings through NVML. It adds driver-load and readiness checks, locked state-file persistence, allocator gating, and decision logic based on Kubernetes evidence and GPU pods. Tests cover parsing, setting application, state transitions, gate behavior, and decision outcomes.

Merge Risk: Low · up to 0f881

This adds GPU configuration parsing, NVML setting application, and allocator gating that are not yet wired into the operator, so there is no immediate production impact. Before this logic is enabled, fix three issues. A failed gate or toolkit-validation container is currently treated as successful completion. Merged YAML entries can bypass the entry-name validation. One decision test can pass despite an unexpected error. The change is safe to merge with these follow-ups tracked.

Security architecture risk: Minimal · up to 0f881

The inspected change does not establish an introduced security weakness or broken coordination guarantee. Configuration and allocator release share an exclusive lock, and release is recorded before proceeding. Production activation is deferred; configuration permissions and deployment integration still require assessment when connected.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — Once integrated with sufficient NVML authority, one decision can affect settings across all GPUs enumerated by its local NVML session, including pending ECC mode. The source does not establish tenant-scoped authorization or cluster-wide document distribution, so neither tenant isolation nor a cluster-wide attacker reach is verified.

Trust Boundaries and Controls

  • observed — The shared state mediates the allocator-to-configuration boundary. Gates record a pass before returning; decisions reject current-load passes and passes made without a decision. Earlier-load releases trigger additional allocator and workload checks. Failed gate API reads treat the manager as expected rather than authorizing immediate release.
  • observed — The document enters through an injected snapshot callback. Its ConfigMap read path and writer authorization remain unresolved; parsing is validation, not proof of source authority. No concrete attacker-to-document-to-GPU path is established by this implementation alone.

Resilience and Maintainability Implications

  • observed — Release without a decision is an explicit policy when no manager is expected or the no-module bound expires. It still requires a successfully persisted pass, which later decisions consume as exclusion evidence. Driver-load rechecks and conservative leftover pass records contain reload races; gates release their locks before polling or sleeping.

Hardening Proposals

  • proposed — At production integration, explicitly constrain ConfigMap writers and shared-state directory writers, verify that every participant uses the same lock and boot-scoped storage, and define operator recovery for partial GPU application before enabling allocator gating. These are integration proposals, not observed vulnerabilities.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 1

internal/gpuconfig/decision/decide_test.go: 1

669-673: Minor · Functional Correctness · Quick win

Reject unexpected errors in success cases.

If a success case returns None and an error before writing state, this branch can pass and return. It never checks the expected outcome or persisted configuration.

When neither wantFatal nor wantErrIs expects an error, call require.NoError before this branch.

As per path instructions, flag “assertions that would still pass if the behavior under test were broken”.

Source: Path instructions

Nitpick comments: 1

internal/gpuconfig/apply/nvmllib/nvmllib_test.go: 1

82-83: Trivial · Maintainability & Code Quality · Quick win

Check the device returned by a successful lookup.

A DeviceByIndex(0) implementation that returns nil, nil would pass this assertion. Keep the returned device and check that UUID() returns the identity configured on dev. This verifies that callers receive a usable wrapper for the selected device.

As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”

Source: Path instructions

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 246a9ca and 0f88146.

Files ignored due to path filters: 39

  • go.sum is excluded by !**/*.sum, !go.sum
  • vendor/github.com/NVIDIA/go-nvml/LICENSE is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_linux.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_other.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/api.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers.h is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers_static.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const_static.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/device.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/doc.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/dynamicLibrary_mock.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/event_set.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/gpm.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/init.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/lib.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/memory_limits.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/README.md is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/computeinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/device.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/eventset.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/extendedinterface.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpmsample.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpuinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/interface.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/unit.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgpuinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgputypeid.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.h is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/refcount.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/return.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/system.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/types_gen.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/unit.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/vgpu.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/zz_generated.api.go is excluded by !vendor/**, !**/zz_generated.*.go
  • vendor/modules.txt is excluded by !vendor/**
Files selected for processing: 28

  • go.mod
  • internal/gpuconfig/apply/apply.go
  • internal/gpuconfig/apply/apply_test.go
  • internal/gpuconfig/apply/fake/fake.go
  • internal/gpuconfig/apply/nvml.go
  • internal/gpuconfig/apply/nvmllib/nvmllib.go
  • internal/gpuconfig/apply/nvmllib/nvmllib_test.go
  • internal/gpuconfig/config/document.go
  • internal/gpuconfig/config/document_test.go
  • internal/gpuconfig/config/match.go
  • internal/gpuconfig/config/match_test.go
  • internal/gpuconfig/config/schema.go
  • internal/gpuconfig/decision/decide.go
  • internal/gpuconfig/decision/decide_test.go
  • internal/gpuconfig/decision/evidence.go
  • internal/gpuconfig/decision/helpers_test.go
  • internal/gpuconfig/decision/pods.go
  • internal/gpuconfig/gate/gate.go
  • internal/gpuconfig/gate/gate_test.go
  • internal/gpuconfig/loadid/driver.go
  • internal/gpuconfig/loadid/loadid.go
  • internal/gpuconfig/loadid/loadid_test.go
  • internal/gpuconfig/statefile/file.go
  • internal/gpuconfig/statefile/file_test.go
  • internal/gpuconfig/statefile/state.go
  • internal/gpuconfig/statefile/state_test.go
  • internal/gpuconfig/status/sync.go
  • internal/gpuconfig/status/sync_test.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 5 remain after this review.

// stringify, such as null, which it reports unreadably.
func checkKeys(data []byte) error {
var doc yamlv2.MapSlice
if err := yamlv2.Unmarshal(data, &doc); err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Data Integrity & Integration · Quick win

Validate merged entry names before JSON key conversion.

In YAML v2.4.4, MapSlice drops merged items when mappingSlice replaces its output slice. Thus walkKeys cannot inspect entry names supplied through a merge. (raw.githubusercontent.com)

For example:

version: v1
gpus:
  <<:
    1: {deviceSelectors: [{productName: ["*"]}], settings: {powerLimit: 300}}
    "1": {deviceSelectors: [{productName: ["*"]}], settings: {powerLimit: 400}}

The strict conversion accepts the distinct YAML keys, then collapses both to JSON key "1". This bypasses the non-string entry-name check. Resolve sees one entry, so it cannot detect the conflicting settings. If both limits are supported, the surviving limit can be applied. (raw.githubusercontent.com)

Walk a representation that retains merged keys and their original types. Reject this document while preserving valid string-key merges. Add this regression case.

Comment thread internal/gpuconfig/decision/pods.go Outdated
return false
}
st := statuses(pod)
if s, ok := st[toolkitContainer]; ok && s.State.Terminated != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Functional Correctness · Quick win

Require successful termination of the wait container.

On a reload, a gpu-config container that exits with ExitCode=1 satisfies this check even if all later containers still wait. evidence then returns a skip reason. A failed toolkit-validation container causes the same false-positive at Line 129. Termination alone does not establish successful init-container completion. (kubernetes.io)

Require ExitCode == 0 for these two completion checks. Preserve the separate evidence from later containers. Add regression cases for failed waits.

Also applies to: 162-162

Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
@henry118 henry118 changed the title gpu-config: add the gate that holds GPU allocators gpu-config [7/13]: add the gate that holds GPU allocators Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant