Skip to content

gpu-config [6/13]: decide when it is safe to apply GPU settings - #3252

Open
henry118 wants to merge 6 commits into
NVIDIA:mainfrom
henry118:gpu-config/06-decision
Open

henry118 wants to merge 6 commits into
NVIDIA:mainfrom
henry118:gpu-config/06-decision

Conversation

@henry118

@henry118 henry118 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Part of #3246 (6/13). Depends on: #3247, #3250, #3251. Review only the last commit.

Adds internal/gpuconfig/decision: the once-per-driver-load decision to
apply or skip. This is the safety-critical part. Not used yet.


Devin Review

@henry118
henry118 requested a review from a team as a code owner November 17, 2026 18:53
devin-ai-integration[bot]

This comment was marked as resolved.

@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

The changes add GPU configuration parsing and matching, state-file storage, driver readiness checks, and NVML-backed setting application. A decision flow checks driver and cluster evidence, then records whether configuration was applied or skipped. The new tests cover parsing, matching, state operations, NVML calls, and decision outcomes.

Merge Risk: Low · up to d71d0

This change adds the GPU configuration decision flow but does not use it yet, so there is no immediate production impact. One error path returns a "driver reloaded" outcome alongside a transient read error, and a few tests are looser than they should be. Both are worth fixing before the flow is wired in, but neither blocks merging.

Security architecture risk: Minimal · up to d71d0

The change warrants design review because it controls when GPU settings may change. It is not activated in running deployments, and the inspected paths establish no introduced security or failure-containment defect.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — When integrated with sufficient device privileges, one decision can configure every selected GPU visible to its NVML session, affecting workloads sharing those GPUs rather than only one pod or tenant. Current deployed reachability was not identified.

Trust Boundaries and Controls

  • observed — The gate treats prior gate passes, toolkit readiness, advertised GPU capacity, relevant ResourceSlices, and GPU pods as evidence against changing settings. Required read errors stop application. Applier exclusion is scoped by namespace and controller DaemonSet name; unresolved claim references are conservatively classified as GPU-related.

Resilience and Maintainability Implications

  • observed — Driver readiness is checked before opening NVML, and load identity is rechecked after device enumeration. Unsupported or unreadable virtualization modes produce a recorded skip without changing settings.

Hardening Proposals

  • proposed — Before activation, verify that production wiring supplies trusted node and namespace identity, implements the stated read-freshness contracts, and makes allocator-release gates participate in the same lock protocol.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 2

internal/gpuconfig/decision/decide.go: 1

187-189: Minor · Functional Correctness · Quick win

Return None when loadChanged fails.

When d.LoadID() returns an error, loadChanged returns (false, ErrTransient-wrapped err). Line 188 and line 381 still return Result{Outcome: LoadChanged} together with that error. The Outcome doc says None is "the outcome returned with an error". A caller that reads Outcome before err treats a transient read failure as a driver reload and rebinds. Return a zero Result when err != nil.

Proposed fix

-	if changed, err := d.loadChanged(loadID); changed || err != nil {
-		return Result{Outcome: LoadChanged}, err
-	}
+	if changed, err := d.loadChanged(loadID); err != nil {
+		return Result{}, err
+	} else if changed {
+		return Result{Outcome: LoadChanged}, nil
+	}

Also applies to: 380-382

internal/gpuconfig/decision/decide_test.go: 1

669-669: Minor · Functional Correctness · Quick win

Reject unexpected errors in success cases.

If the document-application case returns Result{} and an unexpected error before writing state, this branch accepts None, unchanged state and closed NVML. The subtest then passes without checking Applied.

Enter this branch only when the case expects an error. Otherwise, assert require.NoError(t, err) before checking the outcome.

As per path instructions, flag “assertions that would still pass if the behavior under test were broken.”

Source: Path instructions

Nitpick comments: 3

internal/gpuconfig/statefile/file_test.go: 1

219-220: Trivial · Maintainability & Code Quality · Quick win

Exercise a failed reread after a successful read.

This test starts with l.base == nil. It still passes if Read incorrectly retains the base after a later read fails.

First complete a successful read. Then corrupt the file and repeat the read. Assert that WriteGatePass fails and leaves the corrupt bytes unchanged. This covers the invalidation contract; the supplied implementation already satisfies it.

As per path instructions, “Flag assertions that would still pass if the behavior under test were broken.”

Source: Path instructions

internal/gpuconfig/loadid/loadid_test.go: 1

45-48: Trivial · Maintainability & Code Quality · Quick win

Test load-ID lifetime instead of only repeating the inode lookup.

The expected value uses the same os.Stat and Stat_t.Ino operations as Read. An implementation that caches the first successful ID for a path would pass this test.

Assert that repeated reads return the same ID. Then rename the original directory aside, create a new directory at the original path, and assert that the ID changes. Keeping the original directory prevents inode reuse in this fixture.

As per path instructions, “Flag tests that mirror the implementation instead of checking behavior.”

Source: Path instructions

internal/gpuconfig/decision/decide_test.go: 1

132-132: Trivial · Maintainability & Code Quality · Quick win

Check that evidence-based skips prevent setting writes.

Initialized == false also holds after NVML opens and shuts down. These cases supply no configuration document, so they cannot detect accidental application of configured settings before returning Skipped.

Supply a document with docYAML and assert that e.log.Writes stays empty in the skip cases. Keep the existing outcome and state assertions.

As per path instructions, flag “assertions that would still pass if the behavior under test were broken.”

Source: Path instructions

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 246a9ca and d71d0db.

Files ignored due to path filters: 39

  • go.sum is excluded by !**/*.sum, !go.sum
  • vendor/github.com/NVIDIA/go-nvml/LICENSE is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_linux.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/dl/dl_other.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/api.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers.h is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/cgo_helpers_static.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/const_static.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/device.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/doc.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/dynamicLibrary_mock.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/event_set.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/gpm.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/init.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/lib.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/memory_limits.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/README.md is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/computeinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/device.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/eventset.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/extendedinterface.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpmsample.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/gpuinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/interface.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/unit.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgpuinstance.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/mock/vgputypeid.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/nvml.h is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/refcount.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/return.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/system.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/types_gen.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/unit.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/vgpu.go is excluded by !vendor/**
  • vendor/github.com/NVIDIA/go-nvml/pkg/nvml/zz_generated.api.go is excluded by !vendor/**, !**/zz_generated.*.go
  • vendor/modules.txt is excluded by !vendor/**
Files selected for processing: 26

  • go.mod
  • internal/gpuconfig/apply/apply.go
  • internal/gpuconfig/apply/apply_test.go
  • internal/gpuconfig/apply/fake/fake.go
  • internal/gpuconfig/apply/nvml.go
  • internal/gpuconfig/apply/nvmllib/nvmllib.go
  • internal/gpuconfig/apply/nvmllib/nvmllib_test.go
  • internal/gpuconfig/config/document.go
  • internal/gpuconfig/config/document_test.go
  • internal/gpuconfig/config/match.go
  • internal/gpuconfig/config/match_test.go
  • internal/gpuconfig/config/schema.go
  • internal/gpuconfig/decision/decide.go
  • internal/gpuconfig/decision/decide_test.go
  • internal/gpuconfig/decision/evidence.go
  • internal/gpuconfig/decision/helpers_test.go
  • internal/gpuconfig/decision/pods.go
  • internal/gpuconfig/loadid/driver.go
  • internal/gpuconfig/loadid/loadid.go
  • internal/gpuconfig/loadid/loadid_test.go
  • internal/gpuconfig/statefile/file.go
  • internal/gpuconfig/statefile/file_test.go
  • internal/gpuconfig/statefile/state.go
  • internal/gpuconfig/statefile/state_test.go
  • internal/gpuconfig/status/sync.go
  • internal/gpuconfig/status/sync_test.go

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 6 remain after this review.

Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
Signed-off-by: Henry Wang <henrwang@nvidia.com>
@henry118
henry118 force-pushed the gpu-config/06-decision branch from 74e26ac to 757dad0 Compare November 19, 2026 00:16
Signed-off-by: Henry Wang <henrwang@nvidia.com>
@henry118
henry118 force-pushed the gpu-config/06-decision branch from 757dad0 to f0be296 Compare November 19, 2026 00:50
@henry118 henry118 changed the title gpu-config: decide when it is safe to apply GPU settings gpu-config [6/13]: decide when it is safe to apply GPU settings Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant